Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_ada.ksh @ 1272

Last change on this file since 1272 was 1272, checked in by mafoipsl, 9 years ago
Add a system function IGCM_sys_atlas specially adapted on curie (bug ksh) and allowing error discovery for all systems. Use IGCM_sys_atlas funcion in AA_atlas jobs.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 41.8 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sébastien Denvil
5	# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Ada
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=ada
71	# project name
72	typeset PROJECT=$(echo ${LOADL_STEP_GROUP:=NONE})
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=$( TZ=utc date -d '1970-01-01 '${wall_clock_limit} +%s )
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=llsubmit}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=ergon
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Set environment tools (ferret, nco, cdo, rebuild, ...)
93	#====================================================
94	source /smplocal/pub/Modules/default/init/ksh
95	. /smphome/rech/psl/rpsl035/.atlas_env_ada_intel_2013_0_bash > /dev/null 2>&1
96	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
97	[ ! X${TaskType} = Xchecking ] && module list
98
99	export PATH=${PATH}:/smphome/rech/psl/rpsl035/AddNoise/src_X64_ADA/bin
100	export PATH=${PATH}:/smphome/rech/psl/rpsl035/AddPerturbation/src_X64_ADA/bin
101	export PATH=${PATH}:/smphome/rech/psl/rpsl035/bin/
102
103	#====================================================
104	# Host specific DIRECTORIES
105	#====================================================
106
107	#====================================================
108	#- MirrorlibIGCM for frontend
109	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
110
111	#====================================================
112	#- libIGCM_POST for frontend
113	typeset -r libIGCM_POST=${libIGCM}
114
115	#====================================================
116	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
117	typeset -r R_EXE="${MODIPSL}/bin"
118
119	#====================================================
120	#- SUBMIT_DIR : submission dir
121	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${LOADL_STEP_INITDIR}}
122
123	#====================================================
124	#- IN
125	typeset -r R_IN=${R_IN:=/workgpfs/rech/psl/rpsl035/IGCM}
126	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/u/rech/psl/rpsl376}
127
128	#====================================================
129	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
130	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${TMPDIR}}
131
132	#====================================================
133	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
134	typeset -r OUTCOMMAND_PATH=${RUN_DIR_PATH:-/tmp}
135
136	#====================================================
137	#- HOST_MPIRUN_COMMAND
138	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time poe"}
139
140	#====================================================
141	#- Max number of arguments passed to nco operator or demigration command
142	UNIX_MAX_LIMIT=360
143
144	#====================================================
145	#- set PackDefault to true on ada
146	PackDefault=true
147
148	#====================================================
149	#- Number of core per node (max number of OpenMP task)
150	NUM_COREPERNODE=8
151
152	#====================================================
153	#- Default number of MPI task for IPSL coupled model
154	#- required for backward compatibility
155	#-
156	DEFAULT_NUM_PROC_OCE=5
157	DEFAULT_NUM_PROC_CPL=1
158	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
159
160	#D-#==================================================
161	#D-function IGCM_sys_defineArchives
162	#D-* Purpose:
163	#D-* Define ARCHIVE : Dedicated to large files
164	#D-* Define STORAGE : Dedicated to small/medium files
165	#D-* Define R_OUT : Output tree located on ARCHIVE
166	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
167	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
168	#D-* if SpaceName=TEST everything is stored on WORKDIR
169	#D-* Examples:
170	#D-
171	function IGCM_sys_defineArchives {
172	IGCM_debug_PushStack "IGCM_sys_defineArchives"
173
174	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
175	#====================================================
176	#- ARCHIVE (dedicated to large files)
177	ARCHIVE=${config_UserChoices_ARCHIVE}
178	else
179	#====================================================
180	#- ARCHIVE (dedicated to large files)
181	ARCHIVE=$(echo ${HOME} \| sed -e "s,/.*home/,/arch/home/,")
182	fi
183
184	if [ ! X${config_UserChoices_STORAGE} = X ]; then
185	#====================================================
186	#- STORAGE (dedicated to small/medium files)
187	STORAGE=${config_UserChoices_STORAGE}
188	else
189	#====================================================
190	#- STORAGE (dedicated to small/medium files)
191	STORAGE=${WORKDIR}
192	fi
193
194	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
195	#====================================================
196	#- R_OUT
197	R_OUT=${WORKDIR}/IGCM_OUT
198
199	#====================================================
200	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
201	R_FIG=${WORKDIR}/IGCM_OUT
202
203	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
204
205	else
206	#====================================================
207	#- R_OUT
208	R_OUT=${ARCHIVE}/IGCM_OUT
209
210	#====================================================
211	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
212	R_FIG=${ARCHIVE}/IGCM_OUT
213	fi
214
215	#====================================================
216	#- R_BUF (ONLY FOR double copy an scratch)
217	R_BUF=${WORKDIR}/IGCM_OUT
218
219	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
220	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
221	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
222
223	IGCM_debug_PopStack "IGCM_sys_defineArchives"
224	}
225
226	#D-#==================================================
227	#D-function IGCM_sys_RshArchive
228	#D-* Purpose: Archive rsh command
229	#D-* Examples:
230	#D-
231	function IGCM_sys_RshArchive {
232	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
233	DEBUG_sys=false IGCM_sys_IsFileArchived "$@"
234	if [ $? = 0 ] ; then
235	rsh ${STOREHOST} exec /bin/ksh <<-EOF
236	${@}
237	EOF
238	status=$?
239	else
240	/bin/ksh <<-EOF
241	${@}
242	EOF
243	status=$?
244	fi
245	if [ ${status} -gt 0 ] ; then
246	IGCM_debug_Print 2 "IGCM_sys_RshArchive : rsh or command failed error code ${status}"
247	IGCM_debug_Exit "IGCM_sys_RshArchive"
248	fi
249	IGCM_debug_PopStack "IGCM_sys_RshArchive"
250	}
251
252	#D-#==================================================
253	#D-function IGCM_sys_RshArchive_NoError
254	#D-* Purpose: Archive rsh command, without error
255	#D-* used only in monitoring.job
256	#D-* Examples:
257	#D-
258	function IGCM_sys_RshArchive_NoError {
259	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
260	DEBUG_sys=false IGCM_sys_IsFileArchived "$@"
261	if [ $? = 0 ] ; then
262	rsh ${STOREHOST} exec /bin/ksh <<-EOF
263	${@} 2> /dev/null
264	EOF
265	else
266	/bin/ksh <<-EOF
267	${@} 2> /dev/null
268	EOF
269	fi
270	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
271	}
272
273	#D-#==================================================
274	#D-function IGCM_sys_MkdirArchive
275	#D-* Purpose: Mkdir on Archive
276	#D-* Examples:
277	#D-
278	function IGCM_sys_MkdirArchive {
279	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
280	if ( $DEBUG_sys ) ; then
281	echo "IGCM_sys_MkdirArchive :" $@
282	fi
283	#- creation de repertoire sur le serveur fichier
284	DEBUG_sys=false IGCM_sys_IsFileArchived $1
285	if [ $? = 0 ] ; then
286	rsh ${STOREHOST} -n mkdir -p $1
287	status=$?
288	else
289	mkdir -p $1
290	status=$?
291	fi
292
293	if [ ${status} -gt 0 ] ; then
294	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : rsh or mkdir failed error code ${status}"
295	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
296	fi
297	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
298	}
299
300	#D-#==================================================
301	#D-function IGCM_sys_TestDirArchive
302	#D-* Purpose: Test Directory that must exists on Archive
303	#D-* Examples:
304	#D-
305	function IGCM_sys_TestDirArchive {
306	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
307	if ( $DEBUG_sys ) ; then
308	echo "IGCM_sys_TestDirArchive :" $@
309	fi
310	typeset ExistFlag
311	#Command depends on targeted file system
312	DEBUG_sys=false IGCM_sys_IsFileArchived $1
313	if [ $? = 0 ] ; then
314	ExistFlag=$( IGCM_sys_RshArchive "[ -d $1 ] && echo 0 \|\| echo 1" )
315	else
316	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
317	fi
318	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
319	return ${ExistFlag}
320	}
321
322	#D-#==================================================
323	#D-function IGCM_sys_IsFileArchived
324	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
325	#D-* Examples:
326	#D-
327	function IGCM_sys_IsFileArchived {
328	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
329	if ( $DEBUG_sys ) ; then
330	echo "IGCM_sys_IsFileArchived :" $@
331	fi
332	typeset IsArchivedFlag
333	IsArchivedFlag=$( [ "X$( echo $@ \| grep \/arch\/home )" != "X" ] && echo 0 \|\| echo 1 )
334	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
335
336	return ${IsArchivedFlag}
337	}
338
339	#D-#==================================================
340	#D-function IGCM_sys_TestFileArchive
341	#D-* Purpose: Test file that must NOT EXISTS on Archive
342	#D-* Examples:
343	#D-
344	function IGCM_sys_TestFileArchive {
345	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
346	typeset ExistFlag
347	ExistFlag=$( IGCM_sys_RshArchive "[ -f $1 ] && echo 0 \|\| echo 1" )
348	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
349
350	return ${ExistFlag}
351	}
352
353	#D-#==================================================
354	#D-function IGCM_sys_CountFileArchive
355	#D-* Purpose: Count files on Archive filesystem
356	#D-* Examples:
357	#D-
358	function IGCM_sys_CountFileArchive {
359	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
360	#Command depends on targeted file system
361	DEBUG_sys=false IGCM_sys_IsFileArchived $1
362	if [ $? = 0 ] ; then
363	IGCM_sys_RshArchive "ls ${@} 2>/dev/null \| wc -l"
364	status=$?
365	else
366	ls ${@} 2>/dev/null \| wc -l
367	status=$?
368	fi
369	if [ ${status} -gt 0 ] ; then
370	echo "IGCM_sys_CountFileArchive : erreur."
371	fi
372	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
373	}
374
375	#D-#==================================================
376	#D-function IGCM_sys_Tree
377	#D-* Purpose: Tree directories with files on ${ARCHIVE}
378	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
379	#D-
380	function IGCM_sys_Tree {
381	IGCM_debug_PushStack "IGCM_sys_Tree" $@
382	if ( $DEBUG_sys ) ; then
383	echo "IGCM_sys_Tree :" $@
384	fi
385
386	\mfls -R $@
387
388	IGCM_debug_PopStack "IGCM_sys_Tree"
389	}
390
391	#D-#==================================================
392	#D-function IGCM_sys_Qsub
393	#D-* Purpose: Qsub new job
394	#D-* Examples:
395	#D-
396	function IGCM_sys_Qsub {
397	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
398	if ( $DEBUG_sys ) ; then
399	echo "IGCM_sys_Qsub :" $@
400	fi
401	typeset status
402
403	# We have to change output/error file
404	[ ${#@} = 1 ] && REP_FOR_JOB=${SUBMIT_DIR}
405	[ ${#@} = 2 ] && REP_FOR_JOB=${2}
406	sed -e "s:\# \@ output = .:\# \@ output = ${Script_Output}:" \
407	-e "s:\# \@ error = .:\# \@ error = ${Script_Output}:" \
408	$1 > ${REP_FOR_JOB}/JOB_FOR_IGCM
409	cd $REP_FOR_JOB ; /usr/bin/llsubmit JOB_FOR_IGCM > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1 ; status=$? ; cd - ;
410
411	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
412	if [ ${status} -gt 0 ] ; then
413	IGCM_debug_Print 2 "IGCM_sys_Qsub $1 : error code ${status}"
414	IGCM_debug_Exit "IGCM_sys_Qsub"
415	else
416	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ \| tail -1 \| sed -e s/\"//g )
417	IGCM_sys_Rm ${REP_FOR_JOB}/JOB_FOR_IGCM
418	fi
419	IGCM_debug_PopStack "IGCM_sys_Qsub"
420	}
421
422	#D-#==================================================
423	#D-function IGCM_sys_QsubPost
424	#D-* Purpose: Qsub new job on scalaire
425	#D-* Examples:
426	#D-
427	function IGCM_sys_QsubPost {
428	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
429	if ( $DEBUG_sys ) ; then
430	echo "IGCM_sys_QsubPost :" $@
431	fi
432	typeset status
433
434	# We have to change output/error file
435	[ ${#@} = 1 ] && REP_FOR_JOB=${POST_DIR}
436	[ ${#@} = 2 ] && REP_FOR_JOB=${2}
437
438	sed -e "s:\# \@ output = .:\# \@ output = ${Script_Post_Output}.out:" \
439	-e "s:\# \@ error = .:\# \@ error = ${Script_Post_Output}.out:" \
440	${libIGCM_POST}/$1.job > ${REP_FOR_JOB}/JOB_FOR_IGCM_$$
441
442	cd $REP_FOR_JOB ; /usr/bin/llsubmit JOB_FOR_IGCM_$$ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1 ; status=$? ; cd - ;
443
444	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
445	if [ ${status} -gt 0 ] ; then
446	IGCM_debug_Print 2 "IGCM_sys_QsubPost $1 : error code ${status}"
447	IGCM_debug_Exit "IGCM_sys_QsubPost"
448	else
449	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ \| tail -1 \| sed -e s/\"//g )
450	IGCM_sys_Rm ${REP_FOR_JOB}/JOB_FOR_IGCM_$$
451	fi
452	IGCM_debug_PopStack "IGCM_sys_QsubPost"
453	}
454
455	#D-*************************
456	#D- File transfer functions
457	#D-*************************
458	#D-
459
460	#D-#==================================================
461	#D-function IGCM_sys_RmRunDir
462	#D-* Purpose: rm tmpdir (dummy function most of the time batch
463	#D- scheduler will do the job)
464	#D-* Examples:
465	#D-
466	function IGCM_sys_RmRunDir {
467	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
468	if ( $DEBUG_sys ) ; then
469	echo "IGCM_sys_RmRunDir :" $@
470	IGCM_debug_Print 1 "Dummy call, let the scheduler do that."
471	fi
472	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
473	}
474
475	#D-#==================================================
476	#D-function IGCM_sys_Put_Dir
477	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
478	#D-* Examples:
479	#D-
480	function IGCM_sys_Put_Dir {
481	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
482	if ( $DEBUG_sys ) ; then
483	echo "IGCM_sys_Put_Dir :" $@
484	fi
485	if [ $DRYRUN = 0 ]; then
486	if [ ! -d ${1} ] ; then
487	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
488	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
489	return
490	fi
491
492	typeset status
493
494	# Only if we use rsync
495	#IGCM_sys_TestDirArchive $( dirname $2 )
496	#
497	#Command depends on targeted file system
498	DEBUG_sys=false IGCM_sys_IsFileArchived $2
499	if [ $? = 0 ] ; then
500	\rcp -r $1 ${STOREHOST}:$2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
501	status=$?
502	else
503	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
504	status=$?
505	fi
506
507	if [ ${status} -gt 0 ] ; then
508	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : rcp or cp failed error code ${status}"
509	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
510	IGCM_debug_Exit "IGCM_sys_Put_Dir"
511	else
512	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
513	fi
514	fi
515	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
516	}
517
518	#D-#==================================================
519	#D-function IGCM_sys_Get_Dir
520	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
521	#D-* Examples:
522	#D-
523	function IGCM_sys_Get_Dir {
524	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
525	if ( $DEBUG_sys ) ; then
526	echo "IGCM_sys_Get_Dir :" $@
527	fi
528	if [ $DRYRUN = 0 ]; then
529	typeset NB_ESSAI DELAI status i
530	# number of tentative
531	NB_ESSAI=3
532	# time delay between tentative
533	DELAI=2
534
535	# Only if we use rsync
536	#IGCM_sys_TestDirArchive $( dirname $2 )
537	#
538	# Command depends on targeted filesystem
539	DEBUG_sys=false IGCM_sys_IsFileArchived $1
540	if [ $? = 0 ] ; then
541	# add dmget (to demigrate all offline files) to reduce time of this command :
542	#IGCM_sys_RshArchive "dmfind $1 -state MIG -o -state OFL -o -state PAR \| dmget -q -n"
543	i=0
544	while [ $i -lt $NB_ESSAI ] ; do
545	\rcp -rp ${STOREHOST}:$1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
546	status=$?
547	if [ ${status} -gt 0 ]; then
548	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : rcp failed error code ${status} ${i}/${NB_ESSAI}"
549	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
550	sleep $DELAI
551	else
552	break
553	fi
554	(( i = i + 1 ))
555	done
556	else
557	\cp -rp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
558	status=$?
559	if [ ${status} -gt 0 ] ; then
560	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
561	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
562	IGCM_debug_Exit "IGCM_sys_Get_Dir"
563	else
564	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
565	fi
566	fi
567	if [ ${status} -gt 0 ] ; then
568	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : (r)cp failed error code ${status}"
569	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
570	IGCM_debug_Exit "IGCM_sys_Get_Dir"
571	else
572	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
573	fi
574	fi
575	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
576	}
577
578	#D-#==================================================
579	#D-function IGCM_sys_Put_Rest
580	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
581	#D- File and target directory must exist.
582	#D-* Examples:
583	#D-
584	function IGCM_sys_Put_Rest {
585	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
586	if ( $DEBUG_sys ) ; then
587	echo "IGCM_sys_Put_Rest :" $@
588	fi
589	if [ $DRYRUN = 0 ]; then
590	if [ ! -f ${1} ] ; then
591	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
592	IGCM_debug_Exit "IGCM_sys_Put_Rest"
593	fi
594
595	typeset status
596	#
597	if [ X${JobType} = XRUN ] ; then
598	IGCM_sys_Chmod 444 ${1}
599	fi
600	#
601	#
602	#Command depends on targeted file system
603	DEBUG_sys=false IGCM_sys_IsFileArchived $2
604	if [ $? = 0 ] ; then
605	mfput $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
606	status=$?
607	else
608	IGCM_sys_MkdirArchive $( dirname $2 )
609	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
610	status=$?
611	fi
612
613	# #RSYNC WITH NETWORK RSH CALL
614	# IGCM_sys_MkdirArchive $( dirname $2 )
615	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
616	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
617
618	# #RSYNC WITH NFS USE
619	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
620	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
621
622	# status=$?
623	# IGCM_sys_Rsync_out $status
624
625	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
626	# (( status=status+$? ))
627
628	if [ ${status} -gt 0 ] ; then
629	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : mfput or cp failed error code ${status}"
630	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
631	IGCM_debug_Exit "IGCM_sys_Put_Rest"
632	else
633	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
634	fi
635	fi
636	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
637	}
638
639	#D-#==================================================
640	#D-function IGCM_sys_Put_Out
641	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
642	#D-* Examples:
643	#D-
644	function IGCM_sys_Put_Out {
645	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
646	if ( $DEBUG_sys ) ; then
647	echo "IGCM_sys_Put_Out :" $@
648	fi
649
650	typeset status
651
652	if [ $DRYRUN = 0 ]; then
653	if [ ! -f ${1} ] ; then
654	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
655	IGCM_debug_PopStack "IGCM_sys_Put_Out"
656	return 1
657	fi
658	#
659	if [ X${JobType} = XRUN ] ; then
660	if [ X${3} = X ] ; then
661	IGCM_sys_Chmod 444 ${1}
662	fi
663	fi
664	#
665	#
666	#Command depends on targeted file system
667	DEBUG_sys=false IGCM_sys_IsFileArchived $2
668	if [ $? = 0 ] ; then
669	mfput $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
670	status=$?
671	else
672	IGCM_sys_MkdirArchive $( dirname $2 )
673	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
674	status=$?
675	fi
676
677	# #RSYNC WITH NETWORK RSH CALL
678	# IGCM_sys_MkdirArchive $( dirname $2 )
679	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
680	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
681
682	# #RSYNC WITH NFS USE
683	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
684	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
685
686	# status=$?
687	# IGCM_sys_Rsync_out $status
688
689	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
690	# (( status=status+$? ))
691
692	if [ ${status} -gt 0 ] ; then
693	IGCM_debug_Print 2 "IGCM_sys_Put_Out : mfput or cp failed error code ${status}"
694	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
695	IGCM_debug_Exit "IGCM_sys_Put_Out"
696	else
697	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
698	fi
699	fi
700	IGCM_debug_PopStack "IGCM_sys_Put_Out"
701	return 0
702	}
703
704	#D-#==================================================
705	#D-function IGCM_sys_Get
706	#D-* Purpose: Get a file from ${ARCHIVE}
707	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
708	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
709	function IGCM_sys_Get {
710	IGCM_debug_PushStack "IGCM_sys_Get" $@
711
712	typeset DEST status dm_liste
713
714	if ( $DEBUG_sys ) ; then
715	echo "IGCM_sys_Get :" $@
716	fi
717	if [ $DRYRUN -le 2 ]; then
718	if [ X${1} = X'/l' ] ; then
719	eval set +A dm_liste \${${2}}
720	else
721	dm_liste=${1}
722	fi
723	eval DEST=\${${#}}
724
725	# test if the (first) file is present in the old computation :
726	DEBUG_sys=false IGCM_sys_IsFileArchived ${dm_liste[0]}
727	if [ $? = 0 ] ; then
728	IGCM_sys_TestFileArchive ${dm_liste[0]}
729	status=$?
730	else
731	IGCM_sys_TestFileBuffer ${dm_liste[0]}
732	status=$?
733	fi
734
735	if [ ${status} -gt 0 ] ; then
736	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
737	IGCM_debug_Exit "IGCM_sys_Get"
738	return
739	fi
740
741	#Command depends on targeted file system
742	DEBUG_sys=false IGCM_sys_IsFileArchived ${dm_liste[0]}
743	if [ $? = 0 ] ; then
744	mfget ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
745	status=$?
746	else
747	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
748	status=$?
749	fi
750
751	# #RSYNC WITH NETWORK RSH CALL
752	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
753	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
754
755	# #RSYNC WITH NFS USE
756	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
757	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
758
759	# status=$?
760	# IGCM_sys_Rsync_out $status
761
762	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
763	# (( status=status+$? ))
764
765	if [ ${status} -gt 0 ] ; then
766	IGCM_debug_Print 2 "IGCM_sys_Get : mfget or cp failed error code ${status}"
767	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
768	IGCM_debug_Exit "IGCM_sys_Get"
769	else
770	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
771	fi
772	fi
773	IGCM_debug_PopStack "IGCM_sys_Get"
774	}
775
776	#D-#==================================================
777	#D-function IGCM_sys_GetDate_Monitoring
778	#D-* Purpose: get the last year for which the monitoring has been computed
779	#D-* Examples:
780	#D-
781	function IGCM_sys_GetDate_Monitoring {
782	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
783	if ( $DEBUG_sys ) ; then
784	echo "IGCM_sys_GetDate_Monitoring :" $@
785	fi
786
787	IGCM_sys_Cd /tmp
788	# NASTY HACK. We force /tmp here. This function is called only from the front-end and interactively.
789	RUN_DIR_PATH=/tmp IGCM_sys_Get ${1} .
790	eval ${2}=$( cdo showyear $( basename ${1} ) 2> /dev/null \| gawk '{ print $NF }' )
791	# Need to erase this file to avoid collision (permission denied getting file) between users on the front-end
792	IGCM_sys_Rm -f $( basename ${1} )
793	IGCM_sys_Cd -
794
795	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
796	}
797
798	#D-#==================================================
799	#D-function IGCM_sys_Put_Dods
800	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
801	#D-* Examples:
802	#D-
803	function IGCM_sys_Put_Dods {
804	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
805	if ( $DEBUG_sys ) ; then
806	echo "IGCM_sys_Put_Dods :" $@
807	fi
808	typeset status
809	if [ $DRYRUN = 0 ]; then
810	# We take our time on that
811	sleep 10
812	IGCM_sys_TestDirArchive ${R_SAVE}/${1}
813	if [ $? != 0 ] ; then
814	echo "WARNING : IGCM_sys_Put_Dods ${R_SAVE}/${1} DOES NOT EXIST ."
815	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
816	return
817	fi
818
819	rsh ${STOREHOST} exec /bin/ksh <<EOF
820	cd ${R_SAVE}
821	/arch/local/bin/dods_rm DODS/pub/${LOGIN}/${R_DODS}/${1} > /dev/null 2>&1
822	/bin/chmod -R u+w ${R_SAVE}/${1}
823	/arch/local/bin/dods_cp ${1} DODS/pub/${LOGIN}/${R_DODS} > /dev/null 2>&1
824	/bin/chmod -R +rX ${R_SAVE}/${1}
825	/bin/chmod -R u+w ${R_SAVE}/${1}
826	EOF
827	status=$?
828
829	if [ ${status} -gt 0 ] ; then
830	echo "IGCM_sys_Put_Dods : error."
831	IGCM_debug_Exit "IGCM_sys_Put_Dods"
832	fi
833	fi
834	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
835	}
836
837	##############################################################
838	# REBUILD OPERATOR
839
840	#D-#==================================================
841	#D-function IGCM_sys_sync
842	#D-* Purpose: flush buffer on disk (dummy function on Ada)
843	#D-* Examples:
844	#D-
845	function IGCM_sys_sync {
846	IGCM_debug_PushStack "IGCM_sys_sync" $@
847	if ( $DEBUG_sys ) ; then
848	echo "IGCM_sys_sync :" $@
849	echo "Dummy call, let the system do that."
850	fi
851	IGCM_debug_PopStack "IGCM_sys_sync"
852	}
853
854	############################################################
855	# Activate Running Environnment Variables
856
857	#D-#==================================================
858	#D-function IGCM_sys_activ_variables
859	#D-* Purpose: set environement variables prior to execution
860	#D-* Examples:
861	#D-
862	function IGCM_sys_activ_variables {
863	IGCM_debug_PushStack "IGCM_sys_activ_variables"
864	if ( $DEBUG_sys ) ; then
865	echo "IGCM_sys_activ_variables"
866	fi
867
868	# --------------------------------------------------------------------
869	#D- MPI specifications
870	# --------------------------------------------------------------------
871
872	# --------------------------------------------------------------------
873	#D- Other specifications
874	# --------------------------------------------------------------------
875
876	ulimit -s unlimited
877	## to be done only one time
878	## export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/smplocal/pub/NetCDF/4.1.3/lib:/smplocal/pub/HDF5/1.8.9/seq/lib
879	## echo ${LD_LIBRARY_PATH} \| grep -i netcdf >/dev/null 2>&1 \|\| export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/smplocal/pub/NetCDF/4.1.3/lib:/smplocal/pub/HDF5/1.8.9/seq/lib
880	IGCM_debug_Print 1 "set LD_LIBRARY_PATH=${LD_LIBRARY_PATH}"
881
882	export MP_STDOUTMODE=combined
883	IGCM_debug_Print 1 "set MP_STDOUTMODE=${MP_STDOUTMODE}"
884
885	## fix to reduce memory usage. Required since 2014/22/04. On ada IDRIS.
886	#export MP_EUILIBPATH=/smplocal/lib/ibmhpc/pe12012/ppe.pami/gnu/lib64/pami64
887	#IGCM_debug_Print 1 "set MP_EUILIBPATH=${MP_EUILIBPATH}"
888
889	IGCM_debug_PopStack "IGCM_sys_activ_variables"
890	}
891
892	############################################################
893	# Desactivate Running Environnment Variables
894
895	#D-#==================================================
896	#D-function IGCM_sys_desactiv_variables
897	#D-* Purpose: unset environement variables after execution
898	#D-* Examples:
899	#D-
900	function IGCM_sys_desactiv_variables {
901	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
902	if ( $DEBUG_sys ) ; then
903	echo "IGCM_sys_desactiv_variables"
904	fi
905	# --------------------------------------------------------------------
906	#D- MPI specifications
907	# --------------------------------------------------------------------
908
909	# --------------------------------------------------------------------
910	#D- Other specifications
911	# --------------------------------------------------------------------
912
913	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
914	}
915
916	############################################################
917	# Update job headers to be used by the scheduler
918
919	#D-#==================================================
920	#D-function IGCM_sys_updateHeaders
921	#D-* Purpose: Update job headers to be used by the scheduler
922	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
923	#D-
924	function IGCM_sys_updateHeaders {
925	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
926	if ( $DEBUG_sys ) ; then
927	echo "IGCM_sys_updateHeaders"
928	fi
929	typeset file
930	file=$1
931
932	if [ ${executionType} -eq 1 ] ; then
933	# MPMD + MPI
934	sed -e "/::openMPthreads::/d" \
935	-e "s/::JobNumProcTot::/${coreNumber}/" \
936	${file} > ${file}.tmp
937
938	elif [ ${executionType} -eq 2 ] ; then
939	# MPMD + MPI + OMP
940	sed -e "s/::openMPthreads::/${openMPthreads}/" \
941	-e "s/::JobNumProcTot::/${mpiTasks}/" \
942	${file} > ${file}.tmp
943
944	elif [ ${executionType} -eq 3 ] ; then
945	# SPMD + MPI/OMP
946	sed -e "s/::openMPthreads::/${openMPthreads}/" \
947	-e "s/::JobNumProcTot::/${mpiTasks}/" \
948	${file} > ${file}.tmp
949
950	elif [ ${executionType} -eq 4 ] ; then
951	# SPMD + MPI only
952	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
953	-e "/::openMPthreads::/d" \
954	${file} > ${file}.tmp
955
956	elif [ ${executionType} -eq 5 ] ; then
957	# SPMD + OMP only
958	sed -e "s/::openMPthreads::/${openMPthreads}/" \
959	-e "s/@ job_type = parallel/@ job_type = serial/" \
960	-e "/::JobNumProcTot::/d" \
961	${file} > ${file}.tmp
962
963	elif [ ${executionType} -eq 6 ] ; then
964	# SEQUENTIAL THEN
965	sed -e "s/::JobNumProcTot::/1/" \
966	-e "s/@ job_type = parallel/@ job_type = serial/" \
967	-e "/::openMPthreads::/d" \
968	${file} > ${file}.tmp
969
970	fi
971
972	IGCM_sys_Mv ${file}.tmp ${file}
973
974	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
975	}
976
977	############################################################
978	# Build MPI/OMP scripts run file (dummy function)
979
980	#D-#==================================================
981	#D-function IGCM_sys_build_run_file
982	#D-* Purpose: build run file (deprecated)
983	#D-* Examples:
984	#D-
985	function IGCM_sys_build_run_file {
986
987	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
988
989	}
990
991	############################################################
992	# Build MPI/OMP scripts
993
994	#D-#==================================================
995	#D-function IGCM_sys_build_execution_scripts
996	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
997	#D-* Examples:
998	#D-
999	function IGCM_sys_build_execution_scripts
1000	{
1001	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1002	if ( $DEBUG_sys ) ; then
1003	echo "IGCM_sys_build_execution_scripts " $@
1004	fi
1005
1006	EXECUTION=${HOST_MPIRUN_COMMAND}
1007
1008	# MPMD mode
1009	if ( ${OK_PARA_MPMD} ) ; then
1010
1011	if [ -f run_file ] ; then
1012	IGCM_sys_Rm -f run_file
1013	fi
1014	touch run_file
1015
1016	# Build run_file
1017
1018	# First loop on the components for the coupler ie oasis (only if oasis3)
1019	# the coupler ie oasis3 must be the first one
1020	for comp in ${config_ListOfComponents[*]} ; do
1021
1022	eval ExeNameIn=\${config_Executable_${comp}[0]}
1023	eval ExeNameOut=\${config_Executable_${comp}[1]}
1024
1025	# for CPL component only
1026	if [ "X${comp}" = "XCPL" ] ; then
1027
1028	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1029	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1030
1031	if ( ${OK_PARA_MPI} ) ; then
1032	(( mpi_count = 1 ))
1033	until [ ${mpi_count} -gt ${comp_proc_mpi_loc} ] ; do
1034	echo "./${ExeNameOut}" >> run_file
1035	(( mpi_count = mpi_count + 1 ))
1036	done
1037	else
1038	echo "./${ExeNameOut} " >> run_file
1039	fi
1040	fi
1041	done
1042
1043	# Then second loop on the components
1044	for comp in ${config_ListOfComponents[*]} ; do
1045
1046	eval ExeNameIn=\${config_Executable_${comp}[0]}
1047	eval ExeNameOut=\${config_Executable_${comp}[1]}
1048
1049	# Only if we really have an executable for the component and not the coupler ie oasis:
1050	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1051
1052	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1053	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1054
1055	if ( ${OK_PARA_MPI} ) ; then
1056	(( mpi_count = 1 ))
1057	until [ ${mpi_count} -gt ${comp_proc_mpi_loc} ] ; do
1058	if ( ${OK_PARA_OMP} ) ; then
1059	# Check if the number of threads is correct
1060	case ${comp_proc_omp_loc} in
1061	1\|2\|4)
1062	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1063	;;
1064	8\|16)
1065	IGCM_debug_Exit "ERROR with OMP parameters !"
1066	IGCM_debug_Print 2 "Beware : ${comp_proc_omp_loc} is too much for MPMD mode"
1067	IGCM_debug_Print 2 "You will use more CPU than needed"
1068	IGCM_debug_Verif_Exit
1069	;;
1070	*)
1071	IGCM_debug_Exit "ERROR with OMP parameters !"
1072	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1073	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1074	IGCM_debug_Verif_Exit
1075	;;
1076	esac
1077	echo "env OMP_NUM_THREADS=$comp_proc_omp_loc ./${ExeNameOut}" >> run_file
1078	else
1079	echo "./${ExeNameOut}" >> run_file
1080	fi
1081	(( mpi_count = mpi_count + 1 ))
1082	done
1083	else
1084	echo "./${ExeNameOut} " >> run_file
1085	fi
1086	if ( ${OK_PARA_OMP} ) ; then
1087	export KMP_STACKSIZE=200m
1088	fi
1089	fi
1090	done
1091
1092	EXECUTION="${HOST_MPIRUN_COMMAND} -pgmmodel mpmd -cmdfile ./run_file"
1093
1094	IGCM_sys_Chmod u+x run_file
1095	if ( $DEBUG_sys ) ; then
1096	echo "run_file contains : "
1097	cat run_file
1098	fi
1099
1100	# Only one executable (SPMD mode).
1101	else
1102
1103	for comp in ${config_ListOfComponents[*]} ; do
1104
1105	# Only if we really have an executable for the component :
1106	eval ExeNameOut=\${config_Executable_${comp}[1]}
1107	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1108
1109	if ( ${OK_PARA_OMP} ) ; then
1110	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1111	export KMP_STACKSIZE=200m
1112	#export KMP_LIBRARY=turnaround
1113	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1114	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1115	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1116	fi
1117	EXECUTION="${HOST_MPIRUN_COMMAND} ./${ExeNameOut}"
1118	fi
1119
1120	done
1121
1122	fi
1123
1124	IGCM_debug_Print 1 "sys ada : execution command is "
1125	IGCM_debug_Print 1 "$EXECUTION"
1126
1127	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1128	}
1129
1130	#D-#==================================================
1131	#D-function IGCM_sys_check_path
1132	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1133	#D-* do not point to an important use directory. Stop immediately in that case.
1134	#D-* Examples:
1135	#D-
1136	function IGCM_sys_check_path {
1137	IGCM_debug_PushStack "IGCM_sys_check_path"
1138	if ( $DEBUG_sys ) ; then
1139	echo "IGCM_sys_check_path"
1140	fi
1141
1142	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1143	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1144	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1145	IGCM_debug_Exit "This will stop the job"
1146	fi
1147	IGCM_debug_PopStack "IGCM_sys_check_path"
1148	}
1149
1150	#D-#==================================================
1151	#D-function IGCM_sys_check_quota
1152	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1153	#D-* Examples:
1154	#D-
1155	function IGCM_sys_check_quota {
1156	IGCM_debug_PushStack "IGCM_sys_check_quota"
1157	if ( $DEBUG_sys ) ; then
1158	echo "IGCM_sys_check_quota"
1159	fi
1160	# Limit of quota (in %)
1161	limit_quota=90
1162
1163	# Check of the volume
1164	volume_quota=$(quota_u -w 2>/dev/null\| grep 'Quota soft' \| gawk '{print $5}')
1165	if [ ! X${volume_quota} = X ] ; then
1166	quota_volume=${volume_quota%%\%}
1167	# echo $quota_volume
1168	if [ $(echo "${quota_volume} > ${limit_quota}" \| bc) -eq 1 ] ; then
1169	IGCM_debug_Print 1 "Please, check your quota of volume on workgpfs"
1170	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1171	IGCM_debug_Print 1 "Use the quota_u -w command to check"
1172	IGCM_debug_Print 1 "You must have more than 10% available to run"
1173	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1174	IGCM_debug_Verif_Exit
1175	fi
1176
1177	fi
1178	IGCM_debug_PopStack "IGCM_sys_check_quota"
1179	}
1180
1181	#D-#==================================================
1182	#D-function IGCM_sys_getJobSchedulerID
1183	#D-* Purpose: Get the job ID during execution
1184	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1185	#D-
1186	function IGCM_sys_getJobSchedulerID {
1187	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1188	if ( $DEBUG_sys ) ; then
1189	echo "IGCM_sys_getJobSchedulerID"
1190	fi
1191
1192	eval ${1}=$( echo $LOADL_STEP_ID \| awk -F. '{print $4}' )
1193
1194	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1195	}
1196
1197	#D-#==================================================
1198	#D-function IGCM_sys_GetJobID
1199	#D-* Purpose: Get the job ID from the JobName
1200	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1201	#D-
1202	function IGCM_sys_GetJobID {
1203	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1204	if ( $DEBUG_sys ) ; then
1205	echo "IGCM_sys_GetJobID"
1206	fi
1207
1208	# Print only the full (-W) JobID (%id) and JobName (%jn)
1209	ID=$( llq -u $2 -W -f %id %jn \| \
1210	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1211
1212	eval ${3}=${ID}
1213	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1214	}
1215
1216	#D-#==================================================
1217	#D-function IGCM_sys_CountJobInQueue
1218	#D-* Purpose: Count number of users job
1219	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1220	#D-
1221	function IGCM_sys_CountJobInQueue {
1222	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1223	if ( $DEBUG_sys ) ; then
1224	echo "IGCM_sys_CountJobInQueue"
1225	fi
1226
1227	# Print only the full (-W) JobName (%jn)
1228	NbRun=$( llq -W -f %jn \| grep -c "$1" )
1229
1230	eval ${2}=${NbRun}
1231
1232	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1233	}
1234
1235	#D-#==================================================
1236	#D-function IGCM_sys_ListJobInQueue
1237	#D-* Purpose: Produce a list of users jobs
1238	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1239	#D-
1240	function IGCM_sys_ListJobInQueue {
1241	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1242	if ( $DEBUG_sys ) ; then
1243	echo "IGCM_sys_ListJobInQueue"
1244	fi
1245
1246	# With -W option, column width is as large as necessary
1247	set -A JobList $( llq -u $1 -W -f %jn \| head -n -2 \| tail -n +3 \| \
1248	gawk '( $1 != /TS/ && \
1249	$1 !~ /PACK/ && \
1250	$1 !~ /REBUILD/ && \
1251	$1 !~ /pack/ ) \
1252	{ print $1 }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1253
1254	eval set -A ${2} ${JobList[*]}
1255
1256	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1257	}
1258
1259	#D-#==================================================
1260	#D-function IGCM_sys_atlas
1261	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1262	#D-* Examples:
1263	#D-
1264	function IGCM_sys_atlas {
1265	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1266	if ( $DEBUG_sys ) ; then
1267	echo "IGCM_sys_atlas :" $@
1268	fi
1269
1270	typeset status
1271
1272	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1273	status=$?
1274	if [ ${status} -gt 0 ] ; then
1275	echo "IGCM_sys_atlas : error code ${status}"
1276	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1277	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1278	IGCM_debug_PopStack "IGCM_sys_atlas"
1279	return 1
1280	else
1281	IGCM_debug_PopStack "IGCM_sys_atlas"
1282	return 0
1283	fi
1284
1285	IGCM_debug_PopStack "IGCM_sys_atlas"
1286	}
1287

Note: See TracBrowser for help on using the repository browser.

Download in other formats: