Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_curie.ksh @ 1147

Last change on this file since 1147 was 1147, checked in by sdipsl, 9 years ago
broken ensemble. Tentative fix for Agathe. Ensemble are broken since r1057... broken rebuild station due to changes in LMDZ station files. Tentative fix for Lidia.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 93.4 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Curie
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=curie
71	# add default project on curie
72	typeset DEFAULT_PROJECT=gen0826
73
74	#D-
75	#D-#==================================================
76	#D-Program used in libIGCM
77	#D-#==================================================
78
79	# Submit command
80	typeset SUBMIT=${SUBMIT:=ccc_msub}
81	# rsync with path
82	typeset -r RSYNC=/usr/bin/rsync
83	# RSYNC_opt args to rsync
84	typeset -r RSYNC_opt="-va"
85	# ie storage filesystem
86	typeset -r STOREHOST=${MASTER}
87
88	#====================================================
89	# Source default environment
90	#====================================================
91	##. /etc/profile
92
93	#====================================================
94	# Set environment tools (ferret, nco, cdo)
95	#====================================================
96	if [ X${TaskType} = Xcomputing ] ; then
97	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
98	# to run with netcdf 3.6.3 ie compilation done before 17/2/2014
99	# uncomment 2 lines :
100	# module unload netcdf
101	# module load netcdf/3.6.3
102	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddNoise/src_X64_CURIE/bin
103	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddPerturbation/src_X64_CURIE/bin
104	else
105	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
106	fi
107
108	#====================================================
109	# Specific for ocean additionnal diagnostic
110	export FER_GO="$FER_GO /home/cont003/p86denv/IGCM_POST_UTIL/JNL /home/cont003/p86denv/GRAF /home/cont003/p86denv/GRAF/GO"
111	export FER_PALETTE="$FER_PALETTE /home/cont003/p86denv/GRAF/PALET"
112
113	#====================================================
114	# Host specific DIRECTORIES
115	#====================================================
116
117	# ============ CESIUM START ============ #
118
119	#====================================================
120	#- Mirror libIGCM from titane to cesium if needed
121	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
122	#if [ ! ${ROOTSYS} = "home" ] ; then
123	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
124	#else
125	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
126	#fi
127
128	#====================================================
129	#- libIGCM_POST
130	#if ( ${MirrorlibIGCM} ) ; then
131	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
132	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
133	#else
134	# typeset -r libIGCM_POST=${libIGCM}
135	#fi
136
137	# ============ CESIUM END ============ #
138
139	#====================================================
140	#- MirrorlibIGCM for frontend
141	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
142
143	#====================================================
144	#- libIGCM_POST for frontend
145	typeset -r libIGCM_POST=${libIGCM}
146
147	#====================================================
148	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
149	typeset -r R_EXE="${MODIPSL}/bin"
150
151	#====================================================
152	#- SUBMIT_DIR : submission dir
153	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
154
155	#====================================================
156	#- IN
157	typeset -r R_IN=${R_IN:=/ccc/work/cont003/dsm/p86ipsl/IGCM}
158	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/ccc/work/cont003/dsm/p24data}
159
160	#====================================================
161	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
162	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
163
164	#====================================================
165	#- HOST_MPIRUN_COMMAND
166	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1"}
167
168	#====================================================
169	#- Max number of arguments passed to nco operator or demigration command
170	UNIX_MAX_LIMIT=360
171
172	#====================================================
173	#- set PackDefault true on curie
174	PackDefault=true
175
176	#====================================================
177	#- Number of core per node (max number of OpenMP task)
178	NUM_COREPERNODE=8
179
180	#====================================================
181	#- Default number of MPI task for IPSL coupled model
182	#- required for backward compatibility
183	#-
184	DEFAULT_NUM_PROC_OCE=5
185	DEFAULT_NUM_PROC_CPL=1
186	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
187	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
188
189	#D-#==================================================
190	#D-function IGCM_sys_defineArchives
191	#D-* Purpose:
192	#D-* Define ARCHIVE : Dedicated to large files
193	#D-* Define STORAGE : Dedicated to small/medium files
194	#D-* Define R_OUT : Output tree located on ARCHIVE
195	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
196	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
197	#D-* if SpaceName=TEST everything is stored on SCRATCHDIR
198	#D-* Examples:
199	#D-
200	function IGCM_sys_defineArchives {
201	IGCM_debug_PushStack "IGCM_sys_defineArchives"
202
203	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
204	#====================================================
205	#- ARCHIVE (dedicated to large files)
206	ARCHIVE=${config_UserChoices_ARCHIVE}
207	else
208	#====================================================
209	#- ARCHIVE (dedicated to large files)
210	ARCHIVE=${CCCSTOREDIR}
211	fi
212
213	if [ ! X${config_UserChoices_STORAGE} = X ]; then
214	#====================================================
215	#- STORAGE (dedicated to small/medium files)
216	STORAGE=${config_UserChoices_STORAGE}
217	else
218	#====================================================
219	#- STORAGE (dedicated to small/medium files)
220	STORAGE=${CCCWORKDIR}
221	fi
222
223	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
224	#====================================================
225	#- R_OUT
226	R_OUT=${SCRATCHDIR}/IGCM_OUT
227
228	#====================================================
229	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
230	R_FIG=${SCRATCHDIR}/IGCM_OUT
231
232	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
233
234	else
235	#====================================================
236	#- R_OUT
237	R_OUT=${ARCHIVE}/IGCM_OUT
238
239	#====================================================
240	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
241	R_FIG=${STORAGE}/IGCM_OUT
242	fi
243
244	#====================================================
245	#- R_BUF (ONLY FOR double copy an scratch)
246	R_BUF=${SCRATCHDIR}/IGCM_OUT
247
248	#====================================================
249	#- R_TMP (point a space acting as /tmp)
250	R_TMP=/tmp
251
252	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
253	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
254	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
255
256	IGCM_debug_PopStack "IGCM_sys_defineArchives"
257	}
258
259	#D-#==================================================
260	#D-function IGCM_sys_RshMaster
261	#D-* Purpose: Connection to frontend machine.
262	#D-* Examples:
263	#D-
264	function IGCM_sys_RshMaster {
265	IGCM_debug_PushStack "IGCM_sys_RshMaster" $@
266	/bin/ksh <<-EOF
267	export libIGCM=${libIGCM}
268	export DEBUG_debug=${DEBUG_debug}
269	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
270	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
271	${@}
272	EOF
273	if [ $? -gt 0 ] ; then
274	echo "IGCM_sys_RshMaster : erreur."
275	IGCM_debug_Exit "IGCM_sys_RshMaster"
276	fi
277	IGCM_debug_PopStack "IGCM_sys_RshMaster"
278	}
279
280	#D-#==================================================
281	#D-function IGCM_sys_RshArchive
282	#D-* Purpose: Archive rsh command
283	#D-* Examples:
284	#D-
285	function IGCM_sys_RshArchive {
286	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
287	/bin/ksh <<-EOF
288	${@}
289	EOF
290	status=$?
291	if [ ${status} -gt 0 ] ; then
292	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
293	IGCM_debug_Exit "IGCM_sys_RshArchive"
294	fi
295	IGCM_debug_PopStack "IGCM_sys_RshArchive"
296	}
297
298	#D-#==================================================
299	#D-function IGCM_sys_RshArchive_NoError
300	#D-* Purpose: Archive rsh command, without error
301	#D-* used only in monitoring.job
302	#D-* Examples:
303	#D-
304	function IGCM_sys_RshArchive_NoError {
305	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
306	/bin/ksh <<-EOF
307	${@} 2> \dev\null
308	EOF
309	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
310	}
311
312	#D-#==================================================
313	#D-function IGCM_sys_RshPost
314	#D-* Purpose: Post-process rsh command
315	#D-* Examples:
316	#D-
317	function IGCM_sys_RshPost {
318	IGCM_debug_PushStack "IGCM_sys_RshPost" $@
319	if ( $DEBUG_sys ) ; then
320	echo "IGCM_sys_RshPost :" $@
321	fi
322	# keep standard input (stdin) for the loop onto temporary file
323	cat >/tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
324
325	/bin/ksh </tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
326	if [ $? -gt 0 ] ; then
327	echo "IGCM_sys_RshPost : erreur."
328	IGCM_debug_Exit "IGCM_sys_RshPost"
329	fi
330	# delete temporary file
331	\rm /tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
332
333	# ============ FRONTEND END ============ #
334
335	# ============ CESIUM START ============ #
336	# typeset NB_ESSAI DELAI status i
337	# if [ "X$( grep rebuild_from tmp_IGCM_sys_RshPost_$$ )" != "X" ] ; then
338	# #little hack so that rebuild submission is done on titane not an cesium
339	#
340	# libIGCM_POST_sed=$( echo $libIGCM_POST \| sed 's/\//\\\//g' )
341	# POST_DIR_sed=$( echo ${POST_DIR} \| sed 's/\//\\\//g' )
342	# sed "s/IGCM_sys_QsubPost/IGCM_sys_Qsub/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
343	# sed "s/ rebuild_fromWorkdir/ ${libIGCM_POST_sed}\/rebuild_fromWorkdir.job/g" tmp.txt > tmp_IGCM_sys_RshPost_$$
344	# sed "s/ rebuild_fromArchive/ ${libIGCM_POST_sed}\/rebuild_fromArchive.job/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
345	# sed "s/Script_Post_Output=/Script_Output=${POST_DIR_sed}\//g" tmp.txt > tmp_IGCM_sys_RshPost_$$
346	# \mv tmp.txt tmp_IGCM_sys_RshPost_$$
347	#
348	# echo cat tmp_IGCM_sys_RshPost_$$ AFTER
349	# cat tmp_IGCM_sys_RshPost_$$
350	#
351	# /bin/ksh <tmp_IGCM_sys_RshPost_$$
352	# if [ $? -gt 0 ] ; then
353	# echo "IGCM_sys_RshPost : erreur."
354	# IGCM_debug_Exit "IGCM_sys_RshPost"
355	# fi
356	# # delete temporary file
357	# \rm tmp_IGCM_sys_RshPost_$$
358	#
359	# else
360	# # number of tentative
361	# NB_ESSAI=10
362	# # time delay between tentative
363	# DELAI=10
364	# i=0
365	# while [ $i -ne $NB_ESSAI ] ; do
366	# ssh -t titane996 ssh cesium /bin/ksh <tmp_IGCM_sys_RshPost_$$
367	# status=$?
368	# if [ ${status} -ne 0 ];
369	# then
370	# sleep $DELAI
371	# else
372	# break
373	# fi
374	# let i=$i+1
375	# done
376	# # delete temporary file
377	# \rm tmp_IGCM_sys_RshPost_$$
378	#
379	# if [ ${status} -gt 0 ] ; then
380	# echo "IGCM_sys_RshPost : erreur."
381	# IGCM_debug_Exit "IGCM_sys_RshPost"
382	# fi
383	# fi
384
385	# ============ CESIUM END ============ #
386
387	IGCM_debug_PopStack "IGCM_sys_RshPost"
388	}
389
390	#D-#==================================================
391	#D-function IGCM_sys_SendMail
392	#D-* Purpose: Send mail when simulation is over
393	#D-* Examples:
394	#D-
395	function IGCM_sys_SendMail {
396	IGCM_debug_PushStack "IGCM_sys_SendMail" $@
397
398	if ( $DEBUG_sys ) ; then
399	echo "IGCM_sys_SendMail :" $@
400	fi
401
402	if [ X${1} = XAccounting ] ; then
403	status=Accounting
404	mailText=jobAccounting.mail
405	elif ( ${ExitFlag} ) ; then
406	status=failed
407	mailText=jobEnd.mail
408	else
409	status=completed
410	mailText=jobEnd.mail
411	fi
412
413	# Update selected mail template
414	while read -r line; do
415	eval echo $line >> mail.txt ;
416	done < ${libIGCM}/libIGCM_sys/${mailText}
417
418	if [ ! -z ${config_UserChoices_MailName} ] ; then
419	mail -s "${config_UserChoices_JobName} ${status}" ${config_UserChoices_MailName} < mail.txt
420	elif [ -f ~/.forward ] ; then
421	mail -s "${config_UserChoices_JobName} ${status}" $( cat ~/.forward ) < mail.txt
422	fi
423
424	sleep 10
425	rm -f mail.txt
426
427	IGCM_debug_PopStack "IGCM_sys_SendMail"
428	}
429
430	#D-#==================================================
431	#D-function IGCM_sys_Mkdir
432	#D-* Purpose: Master locale mkdir command
433	#D-* Examples:
434	#D-
435	function IGCM_sys_Mkdir {
436	IGCM_debug_PushStack "IGCM_sys_Mkdir" $@
437	if ( $DEBUG_sys ) ; then
438	echo "IGCM_sys_Mkdir :" $@
439	fi
440	if [ ! -d ${1} ]; then
441	\mkdir -p $1
442	if [ $? -gt 0 ] ; then
443	echo "IGCM_sys_Mkdir : erreur."
444	IGCM_debug_Exit "IGCM_sys_Mkdir"
445	fi
446	fi
447	# vérification :
448	if [ ! -d ${1} ] ; then
449	echo "IGCM_sys_Mkdir : erreur."
450	IGCM_debug_Exit "IGCM_sys_Mkdir"
451	fi
452	IGCM_debug_PopStack "IGCM_sys_Mkdir"
453	}
454
455	#D-#==================================================
456	#D-function IGCM_sys_MkdirArchive
457	#D-* Purpose: Mkdir on Archive
458	#D-* Examples:
459	#D-
460	function IGCM_sys_MkdirArchive {
461	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
462	if ( $DEBUG_sys ) ; then
463	echo "IGCM_sys_MkdirArchive :" $@
464	fi
465	#- creation de repertoire sur le serveur fichier
466	if [ ! -d ${1} ]; then
467	\mkdir -p $1
468	status=$?
469
470	if [ ${status} -gt 0 ] ; then
471	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
472	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
473	fi
474	fi
475	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
476	}
477
478	#D-#==================================================
479	#D-function IGCM_sys_MkdirWork
480	#D-* Purpose: Mkdir on Work
481	#D-* Examples:
482	#D-
483	function IGCM_sys_MkdirWork {
484	IGCM_debug_PushStack "IGCM_sys_MkdirWork" $@
485	if ( $DEBUG_sys ) ; then
486	echo "IGCM_sys_MkdirWork :" $@
487	fi
488	#- creation de repertoire sur le serveur fichier
489	if [ ! -d ${1} ]; then
490	\mkdir -p $1
491	if [ $? -gt 0 ] ; then
492	echo "IGCM_sys_MkdirWork : erreur."
493	IGCM_debug_Exit "IGCM_sys_MkdirWork"
494	fi
495	fi
496	IGCM_debug_PopStack "IGCM_sys_MkdirWork"
497	}
498
499	#D-#==================================================
500	#D-function IGCM_sys_Cd
501	#D-* Purpose: master cd command
502	#D-* Examples:
503	#D-
504	function IGCM_sys_Cd {
505	IGCM_debug_PushStack "IGCM_sys_Cd" $@
506	if ( $DEBUG_sys ) ; then
507	echo "IGCM_sys_Cd :" $@
508	fi
509	\cd $1
510	if [ $? -gt 0 ] ; then
511	echo "IGCM_sys_Cd : erreur."
512	IGCM_debug_Exit "IGCM_sys_Cd"
513	fi
514	IGCM_debug_PopStack "IGCM_sys_Cd"
515	}
516
517	#D-#==================================================
518	#D-function IGCM_sys_Chmod
519	#D-* Purpose: Chmod
520	#D-* Examples:
521	#D-
522	function IGCM_sys_Chmod {
523	IGCM_debug_PushStack "IGCM_sys_Chmod" $@
524	if ( $DEBUG_sys ) ; then
525	echo "IGCM_sys_Chmod :" $@
526	fi
527	\chmod $@
528	if [ $? -gt 0 ] ; then
529	echo "IGCM_sys_Chmod : erreur."
530	IGCM_debug_Exit "IGCM_sys_Chmod"
531	fi
532	IGCM_debug_PopStack "IGCM_sys_Chmod"
533	}
534
535	#D-#==================================================
536	#D-function IGCM_sys_FileSize
537	#D-* Purpose: Filesize
538	#D-* Examples:
539	#D-
540	function IGCM_sys_FileSize {
541	IGCM_debug_PushStack "IGCM_sys_FileSize" $@
542
543	typeset sizeF
544	set +A sizeF -- $( ls -la ${1} )
545	if [ $? -gt 0 ] ; then
546	IGCM_debug_Exit "IGCM_sys_FileSize"
547	fi
548	eval ${2}=${sizeF[4]}
549
550	IGCM_debug_PopStack "IGCM_sys_FileSize"
551	}
552
553	#D-#==================================================
554	#D-function IGCM_sys_TestDir
555	#D-* Purpose: Test Directory that must exists
556	#D-* Examples:
557	#D-
558	function IGCM_sys_TestDir {
559	IGCM_debug_PushStack "IGCM_sys_TestDir" $@
560	if ( $DEBUG_sys ) ; then
561	echo "IGCM_sys_TestDir :" $@
562	fi
563	typeset ExistFlag
564	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
565	IGCM_debug_PopStack "IGCM_sys_TestDir"
566
567	return ${ExistFlag}
568	}
569
570	#D-#==================================================
571	#D-function IGCM_sys_TestDirArchive
572	#D-* Purpose: Test Directory that must exists on Archive
573	#D-* Examples:
574	#D-
575	function IGCM_sys_TestDirArchive {
576	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
577	if ( $DEBUG_sys ) ; then
578	echo "IGCM_sys_TestDirArchive :" $@
579	fi
580	typeset ExistFlag
581	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
582	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
583	return ${ExistFlag}
584	}
585
586	#D-#==================================================
587	#D-function IGCM_sys_IsFileArchived
588	#D-* Purpose: Test file that must NOT EXISTS on Archive
589	#D-* Examples:
590	#D-
591	function IGCM_sys_IsFileArchived {
592	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
593	if ( $DEBUG_sys ) ; then
594	echo "IGCM_sys_IsFileArchived :" $@
595	fi
596	typeset IsArchivedFlag
597	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
598	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
599
600	return ${IsArchivedFlag}
601	}
602
603	#D-#==================================================
604	#D-function IGCM_sys_TestFileArchive
605	#D-* Purpose: Test file that must NOT EXISTS on Archive
606	#D-* Examples:
607	#D-
608	function IGCM_sys_TestFileArchive {
609	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
610	typeset ExistFlag
611	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
612	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
613
614	return ${ExistFlag}
615	}
616
617	#D-#==================================================
618	#D-function IGCM_sys_TestFileBuffer
619	#D-* Purpose: Test file that must NOT EXISTS on Buffer
620	#D-* Examples:
621	#D-
622	function IGCM_sys_TestFileBuffer {
623	IGCM_debug_PushStack "IGCM_sys_TestFileBuffer" $@
624	typeset ExistFlag
625	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
626	IGCM_debug_PopStack "IGCM_sys_TestFileBuffer"
627
628	return ${ExistFlag}
629	}
630
631	#D-#==================================================
632	#D-function IGCM_sys_CountFileArchive
633	#D-* Purpose: Count files on Archive filesystem
634	#D-* Examples:
635	#D-
636	function IGCM_sys_CountFileArchive {
637	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
638	ls ${@} 2>/dev/null \| wc -l
639	if [ $? -gt 0 ] ; then
640	echo "IGCM_sys_CountFileArchive : erreur."
641	fi
642	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
643	}
644
645	#D-#==================================================
646	#D-function IGCM_sys_CountFileBuffer
647	#D-* Purpose: Count files on Scratch filesystem
648	#D-* Examples:
649	#D-
650	function IGCM_sys_CountFileBuffer {
651	IGCM_debug_PushStack "IGCM_sys_CountFileBuffer" $@
652	ls ${@} 2>/dev/null \| wc -l
653	if [ $? -gt 0 ] ; then
654	echo "IGCM_sys_CountFileBuffer : erreur."
655	fi
656	IGCM_debug_PopStack "IGCM_sys_CountFileBuffer"
657	}
658
659	#D-#==================================================
660	#D-function IGCM_sys_Tree
661	#D-* Purpose: Tree directories with files on ${ARCHIVE}
662	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
663	#D-
664	function IGCM_sys_Tree {
665	IGCM_debug_PushStack "IGCM_sys_Tree" $@
666	if ( $DEBUG_sys ) ; then
667	echo "IGCM_sys_Tree :" $@
668	fi
669
670	\ls -lR ${@}
671
672	IGCM_debug_PopStack "IGCM_sys_Tree"
673	}
674
675	#D-#==================================================
676	#D-function IGCM_sys_Tar
677	#D-* Purpose: master tar command
678	#D-* Examples:
679	#D-
680	function IGCM_sys_Tar {
681	IGCM_debug_PushStack "IGCM_sys_Tar" $@
682	if ( $DEBUG_sys ) ; then
683	echo "IGCM_sys_Tar :" $@
684	fi
685	\tar cf $@
686	if [ $? -gt 0 ] ; then
687	echo "IGCM_sys_Tar : erreur."
688	IGCM_debug_Exit "IGCM_sys_Tar"
689	fi
690	IGCM_debug_PopStack "IGCM_sys_Tar"
691	}
692
693	#D-#==================================================
694	#D-function IGCM_sys_UnTar
695	#D-* Purpose: master un-tar command
696	#D-* Examples:
697	#D-
698	function IGCM_sys_UnTar {
699	IGCM_debug_PushStack "IGCM_sys_UnTar" $@
700	if ( $DEBUG_sys ) ; then
701	echo "IGCM_sys_UnTar :" $@
702	fi
703	\tar xvf $1
704	if [ $? -gt 0 ] ; then
705	echo "IGCM_sys_UnTar : erreur."
706	IGCM_debug_Exit "IGCM_sys_UnTar"
707	fi
708	IGCM_debug_PopStack "IGCM_sys_UnTar"
709	}
710
711	#D-#==================================================
712	#D-function IGCM_sys_Qsub
713	#D-* Purpose: Qsub new job
714	#D-* Examples:
715	#D-
716	function IGCM_sys_Qsub {
717	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
718	if ( $DEBUG_sys ) ; then
719	echo "IGCM_sys_Qsub :" $@
720	fi
721	typeset options status
722	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
723
724	/usr/bin/ccc_msub ${options} $1 > /tmp/out_command.$$ 2>&1
725	status=$?
726
727	cat /tmp/out_command.$$
728	if [ ${status} -gt 0 ] ; then
729	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
730	IGCM_debug_Exit "IGCM_sys_Qsub"
731	else
732	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
733	fi
734	IGCM_debug_PopStack "IGCM_sys_Qsub"
735	}
736
737	#D-#==================================================
738	#D-function IGCM_sys_QsubPost
739	#D-* Purpose: Qsub new job on scalaire
740	#D-* Examples:
741	#D-
742	function IGCM_sys_QsubPost {
743	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
744	if ( $DEBUG_sys ) ; then
745	echo "IGCM_sys_QsubPost :" $@
746	fi
747	typeset options status
748	MY_PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=${DEFAULT_PROJECT}} \| cut -d@ -f1 )
749
750	# EASIER TO DO THIS INSTEAD OF DUPLICATING libIGCM_sys_curie.ksh
751	case $( hostname -s ) in
752	curie*)
753	options="-Q normal -A ${MY_PROJECT} -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out";;
754	airain*)
755	options="-q ivybridge -A dsm -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
756	esac
757
758	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > /tmp/out_command.$$ 2>&1
759	status=$?
760
761	cat /tmp/out_command.$$
762	if [ ${status} -gt 0 ] ; then
763	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
764	IGCM_debug_Exit "IGCM_sys_QsubPost"
765	else
766	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
767	fi
768	IGCM_debug_PopStack "IGCM_sys_QsubPost"
769	}
770
771	#D-*************************
772	#D- File transfer functions
773	#D-*************************
774	#D-
775
776	#D-#==================================================
777	#D-function IGCM_sys_Rsync_out
778	#D-* Purpose: treat return val of rsync
779	#D-* Examples: IGCM_sys_Rsync_out out_RET_rsync
780	#D- Error values and explanations can depend on your system version.
781	function IGCM_sys_Rsync_out {
782	status=$1
783	if [ ! $status ] ; then
784	echo "rsync error !"
785	fi
786
787	if [ $MYLANG = "fr" ]; then
788	case $status in
789	0) return ;;
790	1) echo "Erreur de rsync ; RERR_SYNTAX : "
791	echo "Erreur de syntaxe ou d'utilisation."
792	return;;
793	2) echo "Erreur de rsync ; RERR_PROTOCOL : "
794	echo "Incompatibilité de protocole."
795	return;;
796	3) echo "Erreur de rsync ; RERR_FILESELECT 3"
797	echo "Erreurs lors de la sélection des fichiers d'entrée sortie et"
798	echo "répertoires"
799	return;;
800	4) echo "Erreur de rsync ; RERR_UNSUPPORTED"
801	echo "Action demandée non supportée : une tentative de manipulation de"
802	echo "fichiers 64-bits sur une plate-forme qui ne les supporte pas a"
803	echo "été faite ; ou une option qui est supportée par le client mais"
804	echo "pas par le serveur a été spécifiée."
805	return;;
806	10) echo "Erreur de rsync ; RERR_SOCKETIO"
807	echo "Erreur dans le socket d'entrée sortie"
808	return;;
809	11) echo "Erreur de rsync ; RERR_FILEIO"
810	echo "Erreur d'entrée sortie fichier"
811	return;;
812	12) echo "Erreur de rsync ; RERR_STREAMIO"
813	echo "Erreur dans flux de donnée du protocole rsync"
814	return;;
815	13) echo "Erreur de rsync ; RERR_MESSAGEIO"
816	echo "Erreur avec les diagnostics du programme"
817	return;;
818	14) echo "Erreur de rsync ; RERR_IPC"
819	echo "Erreur dans le code IPC"
820	return;;
821	20) echo "Erreur de rsync ; RERR_SIGNAL"
822	echo "SIGUSR1 ou SIGINT reçu"
823	return;;
824	21) echo "Erreur de rsync ; RERR_WAITCHILD"
825	echo "Une erreur retournée par waitpid()"
826	return;;
827	22) echo "Erreur de rsync ; RERR_MALLOC"
828	echo "Erreur lors de l'allocation des tampons de mémoire de coeur"
829	return;;
830	23) echo ""
831	echo "Erreur fichier inexistant"
832	return;;
833	30) echo "Erreur de rsync ; RERR_TIMEOUT"
834	echo "Temps d'attente écoulé dans l'envoi/réception de données"
835	return;;
836	*) echo "Erreur de rsync : code de retour de rsync inconnu :" $status
837	return;;
838	esac
839	elif [ $MYLANG = "en" ] ; then
840	case $status in
841	0) return;;
842	1) echo "rsync error : Syntax or usage error "
843	return;;
844	2) echo "rsync error : Protocol incompatibility "
845	return;;
846	3) echo "rsync error : Errors selecting input/output files, dirs"
847	return;;
848	4) echo "rsync error : Requested action not supported: an attempt"
849	echo "was made to manipulate 64-bit files on a platform that cannot support"
850	echo "them; or an option was specified that is supported by the client and"
851	echo "not by the server."
852	return;;
853	5) echo "rsync error : Error starting client-server protocol"
854	return;;
855	10) echo "rsync error : Error in socket I/O "
856	return;;
857	11) echo "rsync error : Error in file I/O "
858	return;;
859	12) echo "rsync error : Error in rsync protocol data stream "
860	return;;
861	13) echo "rsync error : Errors with program diagnostics "
862	return;;
863	14) echo "rsync error : Error in IPC code "
864	return;;
865	20) echo "rsync error : Received SIGUSR1 or SIGINT "
866	return;;
867	21) echo "rsync error : Some error returned by waitpid() "
868	return;;
869	22) echo "rsync error : Error allocating core memory buffers "
870	return;;
871	23) echo "rsync error : Partial transfer due to error"
872	return;;
873	24) echo "rsync error : Partial transfer due to vanished source files"
874	return;;
875	30) echo "rsync error : Timeout in data send/receive "
876	return;;
877	*) echo "rsync error : return code of rsync unknown :" $status
878	return;;
879	esac
880	else
881	echo "unknown language $MYLANG."
882	return
883	fi
884	}
885
886	#D-#==================================================
887	#D-function IGCM_sys_Miror_libIGCM
888	#D-* Purpose: Mirror libIGCM PATH and lib to frontend
889	#D-* Examples:
890	#D-
891	function IGCM_sys_Mirror_libIGCM {
892	IGCM_debug_PushStack "IGCM_sys_Mirror_libIGCM"
893	if ( $DEBUG_sys ) ; then
894	echo "IGCM_sys_Mirror_libIGCM"
895	fi
896
897	typeset status
898
899	mkdir -p ${HOME}/MIRROR/${PATHlibIGCM}
900
901	echo ${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} > /tmp/out_command.$$ 2>&1
902	${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} >> /tmp/out_command.$$ 2>&1
903	status=$?
904
905	if [ ${status} -gt 0 ] ; then
906	echo "IGCM_sys_Mirror_libIGCM Warning : no libIGCM on frontend."
907	cat /tmp/out_command.$$
908	fi
909	IGCM_debug_PopStack "IGCM_sys_Mirror_libIGCM"
910	}
911
912	#D-#==================================================
913	#D-function IGCM_sys_Cp
914	#D-* Purpose: generic cp
915	#D-* Examples:
916	#D-
917	function IGCM_sys_Cp {
918	IGCM_debug_PushStack "IGCM_sys_Cp" $@
919	if ( $DEBUG_sys ) ; then
920	echo "IGCM_sys_Cp :" $@
921	fi
922
923	typeset status
924
925	echo cp $@ > /tmp/out_command.$$ 2>&1
926	\cp $@ >> /tmp/out_command.$$ 2>&1
927	status=$?
928
929	if [ ${status} -gt 0 ] ; then
930	echo "IGCM_sys_Cp : error code ${status}"
931	cat /tmp/out_command.$$
932	IGCM_debug_Exit "IGCM_sys_Cp"
933	else
934	\rm /tmp/out_command.$$
935	fi
936	IGCM_debug_PopStack "IGCM_sys_Cp"
937	}
938
939	#D-#==================================================
940	#D-function IGCM_sys_Rm
941	#D-* Purpose: generic rm
942	#D-* Examples:
943	#D-
944	function IGCM_sys_Rm {
945	IGCM_debug_PushStack "IGCM_sys_Rm" $@
946	if ( $DEBUG_sys ) ; then
947	echo "IGCM_sys_Rm :" $@
948	fi
949
950	typeset status
951
952	echo rm $@ > /tmp/out_command.$$ 2>&1
953	\rm $@ >> /tmp/out_command.$$ 2>&1
954	status=$?
955
956	if [ ${status} -gt 0 ] ; then
957	echo "IGCM_sys_Rm : error code ${status}"
958	cat /tmp/out_command.$$
959	IGCM_debug_Exit "IGCM_sys_Rm"
960	else
961	\rm /tmp/out_command.$$
962	fi
963	IGCM_debug_PopStack "IGCM_sys_Rm"
964	}
965
966	#D-#==================================================
967	#D-function IGCM_sys_RmRunDir
968	#D-* Purpose: rm tmpdir (dummy function most of the time batch
969	#D- scheduler will do the job)
970	#D-* Examples:
971	#D-
972	function IGCM_sys_RmRunDir {
973	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
974	if ( $DEBUG_sys ) ; then
975	echo "IGCM_sys_RmRunDir :" $@
976	fi
977
978	typeset status
979
980	echo rm $@ > /tmp/out_command.$$ 2>&1
981	\rm $@ >> /tmp/out_command.$$ 2>&1
982	status=$?
983
984	if [ ${status} -gt 0 ] ; then
985	echo "IGCM_sys_RmRunDir : rm error code is ${status}."
986	cat /tmp/out_command.$$
987	IGCM_debug_Exit "IGCM_sys_RmRunDir"
988	else
989	\rm /tmp/out_command.$$
990	fi
991	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
992	}
993
994	#D-#==================================================
995	#D-function IGCM_sys_Mv
996	#D-* Purpose: generic move
997	#D-* Examples:
998	#D-
999	function IGCM_sys_Mv {
1000	IGCM_debug_PushStack "IGCM_sys_Mv" $@
1001	if ( $DEBUG_sys ) ; then
1002	echo "IGCM_sys_Mv :" $@
1003	fi
1004
1005	if [ $DRYRUN = 0 ]; then
1006
1007	typeset status
1008
1009	echo mv $@ > /tmp/out_command.$$ 2>&1
1010	\mv $@ >> /tmp/out_command.$$ 2>&1
1011	status=$?
1012
1013	if [ ${status} -gt 0 ] ; then
1014	echo "IGCM_sys_Mv : error code ${status}"
1015	cat /tmp/out_command.$$
1016	IGCM_debug_Exit "IGCM_sys_Mv"
1017	else
1018	\rm /tmp/out_command.$$
1019	fi
1020	fi
1021
1022	IGCM_debug_PopStack "IGCM_sys_Mv"
1023	}
1024
1025	#D-#==================================================
1026	#D-function IGCM_sys_Put_Dir
1027	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
1028	#D-* Examples:
1029	#D-
1030	function IGCM_sys_Put_Dir {
1031	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
1032	if ( $DEBUG_sys ) ; then
1033	echo "IGCM_sys_Put_Dir :" $@
1034	fi
1035	if [ $DRYRUN = 0 ]; then
1036	if [ ! -d ${1} ] ; then
1037	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
1038	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
1039	return
1040	fi
1041
1042	typeset status
1043
1044	# Only if we use rsync
1045	#IGCM_sys_TestDirArchive $( dirname $2 )
1046	#
1047	#USUAL WAY
1048	\cp -r $1 $2 > /tmp/out_command.$$ 2>&1
1049	status=$?
1050
1051	if [ ${status} -gt 0 ] ; then
1052	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
1053	cat /tmp/out_command.$$
1054	IGCM_debug_Exit "IGCM_sys_Put_Dir"
1055	else
1056	\rm /tmp/out_command.$$
1057	fi
1058	fi
1059	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
1060	}
1061
1062	#D-#==================================================
1063	#D-function IGCM_sys_Get_Dir
1064	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
1065	#D-* Examples:
1066	#D-
1067	function IGCM_sys_Get_Dir {
1068	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
1069	if ( $DEBUG_sys ) ; then
1070	echo "IGCM_sys_Get_Dir :" $@
1071	fi
1072	if [ $DRYRUN = 0 ]; then
1073	# if [ ! -d ${1} ] ; then
1074	# echo "WARNING : IGCM_sys_Get_Dir ${1} DOES NOT EXIST ."
1075	# IGCM_debug_PopStack "IGCM_sys_Get_Dir"
1076	# return
1077	# fi
1078
1079	typeset NB_ESSAI DELAI status i
1080	# number of tentative
1081	NB_ESSAI=3
1082	# time delay between tentative
1083	DELAI=2
1084
1085	# Only if we use rsync
1086	#IGCM_sys_TestDirArchive $( dirname $2 )
1087	#
1088	# USUAL WAY
1089	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
1090	ccc_hsm get -r $1
1091
1092	i=0
1093	while [ $i -lt $NB_ESSAI ] ; do
1094	\cp -ur $1 $2 >> /tmp/out_command.$$ 2>&1
1095	status=$?
1096	if [ ${status} -gt 0 ] ; then
1097	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
1098	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
1099	sleep $DELAI
1100	else
1101	break
1102	fi
1103	(( i = i + 1 ))
1104	done
1105
1106	if [ ${status} -gt 0 ] ; then
1107	echo "IGCM_sys_Get_Dir : error."
1108	cat /tmp/out_command.$$
1109	IGCM_debug_Exit "IGCM_sys_Get_Dir"
1110	else
1111	\rm /tmp/out_command.$$
1112	fi
1113	fi
1114	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
1115	}
1116
1117	#D-#==================================================
1118	#D-function IGCM_sys_Get_Master
1119	#D-* Purpose: Copy a complete directory from MASTER filesystem
1120	#D-* Examples:
1121	#D-
1122	function IGCM_sys_Get_Master {
1123	IGCM_debug_PushStack "IGCM_sys_Get_Master" $@
1124	if ( $DEBUG_sys ) ; then
1125	echo "IGCM_sys_Get_Master :" $@
1126	fi
1127	if [ $DRYRUN = 0 ]; then
1128	if ( [ ! -d ${1} ] && [ ! -f ${1} ] ) ; then
1129	echo "WARNING : IGCM_sys_Get_Master ${1} DOES NOT EXIST ."
1130	IGCM_debug_PopStack "IGCM_sys_Get_Master"
1131	return
1132	fi
1133
1134	typeset NB_ESSAI DELAI status i
1135	# number of tentative
1136	NB_ESSAI=3
1137	# time delay between tentative
1138	DELAI=2
1139
1140	i=0
1141	while [ $i -lt $NB_ESSAI ] ; do
1142	\cp -urL $1 $2 > /tmp/out_command.$$ 2>&1
1143	status=$?
1144	if [ ${status} -gt 0 ]; then
1145	IGCM_debug_Print 2 "IGCM_sys_Get_Master : cp failed error code ${status} ${i}/${NB_ESSAI}"
1146	IGCM_debug_Print 2 "IGCM_sys_Get_Master : sleep ${DELAI} seconds and try again."
1147	sleep $DELAI
1148	else
1149	break
1150	fi
1151	(( i = i + 1 ))
1152	done
1153
1154	if [ ${status} -gt 0 ] ; then
1155	echo "IGCM_sys_Get_Master : error."
1156	cat /tmp/out_command.$$
1157	IGCM_debug_Exit "IGCM_sys_Get_Master"
1158	else
1159	\rm /tmp/out_command.$$
1160	fi
1161	fi
1162	IGCM_debug_PopStack "IGCM_sys_Get_Master"
1163	}
1164
1165	#====================================================
1166	#- Call IGCM_sys_Mirror_libIGCM now !
1167	if ( $MirrorlibIGCM ) ; then
1168	IGCM_sys_Mirror_libIGCM
1169	fi
1170
1171	#D-#==================================================
1172	#D-function IGCM_sys_Put_Rest
1173	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
1174	#D- File and target directory must exist.
1175	#D-* Examples:
1176	#D-
1177	function IGCM_sys_Put_Rest {
1178	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
1179	if ( $DEBUG_sys ) ; then
1180	echo "IGCM_sys_Put_Rest :" $@
1181	fi
1182	if [ $DRYRUN = 0 ]; then
1183	if [ ! -f ${1} ] ; then
1184	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
1185	IGCM_debug_Exit "IGCM_sys_Put_Rest"
1186	fi
1187
1188	typeset status
1189	#
1190	# USUAL WAY
1191	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1192	status=$?
1193
1194	# #RSYNC WITH NETWORK SSH CALL
1195	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
1196	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
1197
1198	# #RSYNC WITH NFS USE
1199	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
1200	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
1201
1202	# status=$?
1203	# IGCM_sys_Rsync_out $status
1204
1205	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1206	# (( status=status+$? ))
1207
1208	if [ ${status} -gt 0 ] ; then
1209	echo "IGCM_sys_Put_Rest : cp failed error code ${status}"
1210	[ -f ${1} ] && ls -l ${1}
1211	[ -f ${2} ] && ls -l ${2}
1212	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1213	cat /tmp/out_command.$$
1214	IGCM_debug_Exit "IGCM_sys_Put_Rest"
1215	else
1216
1217	if [ X${JobType} = XRUN ] ; then
1218	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1219	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1220	fi
1221
1222	\rm /tmp/out_command.$$
1223	fi
1224	fi
1225	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
1226	}
1227
1228	#D-#==================================================
1229	#D-function IGCM_sys_PutBuffer_Rest
1230	#D-* Purpose: Put computied restarts on ${SCRATCHDIR}.
1231	#D- File and target directory must exist.
1232	#D-* Examples:
1233	#D-
1234	function IGCM_sys_PutBuffer_Rest {
1235	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Rest" $@
1236	if ( $DEBUG_sys ) ; then
1237	echo "IGCM_sys_PutBuffer_Rest :" $@
1238	fi
1239	if [ $DRYRUN = 0 ]; then
1240	if [ ! -f ${1} ] ; then
1241	echo "ERROR : IGCM_sys_PutBuffer_Rest ${1} DOES NOT EXIST ."
1242	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
1243	fi
1244
1245	typeset status
1246	#
1247	# USUAL WAY
1248	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1249	status=$?
1250
1251	if [ ${status} -gt 0 ] ; then
1252	echo "IGCM_sys_PutBuffer_Rest : error code ${status}"
1253	[ -f ${2} ] && ls -l ${2}
1254	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1255	cat /tmp/out_command.$$
1256	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
1257	else
1258
1259	if [ X${JobType} = XRUN ] ; then
1260	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1261	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1262	fi
1263
1264	\rm /tmp/out_command.$$
1265	fi
1266	fi
1267	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Rest"
1268	}
1269
1270	#D-#==================================================
1271	#D-function IGCM_sys_PrepareTaredRestart
1272	#D-* Purpose: Prepare tared restart to be access by computing job. Identity here.
1273	#D-* Examples:
1274	#D-
1275	function IGCM_sys_PrepareTaredRestart {
1276	IGCM_debug_PushStack "IGCM_sys_PrepareTaredRestart" $@
1277	if [ $DRYRUN = 0 ]; then
1278	[ ! -f $( basename $1 ) ] && IGCM_sys_Get $1 .
1279	fi
1280	IGCM_debug_PopStack "IGCM_sys_PrepareTaredRestart"
1281	}
1282
1283	#D-#==================================================
1284	#D-function IGCM_sys_Put_Out
1285	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
1286	#D-* Examples:
1287	#D-
1288	function IGCM_sys_Put_Out {
1289	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
1290	if ( $DEBUG_sys ) ; then
1291	echo "IGCM_sys_Put_Out :" $@
1292	fi
1293
1294	typeset NB_ESSAI DELAI status i exist skip
1295
1296	# number of tentative
1297	NB_ESSAI=3
1298	# time delay between tentative
1299	DELAI=2
1300
1301	if [ $DRYRUN = 0 ]; then
1302	if [ ! -f ${1} ] ; then
1303	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
1304	IGCM_debug_PopStack "IGCM_sys_Put_Out"
1305	return 1
1306	fi
1307	#
1308	IGCM_sys_MkdirArchive $( dirname $2 )
1309	#
1310
1311	#=====================================================
1312	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
1313	#=====================================================
1314
1315	#echo ${2} \| grep "${R_OUT}" > /dev/null 2>&1
1316	#if [ $? -eq 0 ] ; then
1317	# typeset WORKPATH FILEPATH
1318	# WORKPATH=$( dirname $2 \| sed -e "s\|${R_OUT}\|${R_BUF}\|" )
1319	# IGCM_sys_MkdirWork ${WORKPATH}
1320	# FILEPATH=${WORKPATH}/$( basename $2 )
1321	# #
1322	# IGCM_sys_Cp ${1} ${FILEPATH}
1323	#fi
1324
1325	exist=false
1326	skip=false
1327	if [ -f $2 ] ; then
1328	IGCM_debug_Print 1 "$2 already exist"
1329	ccc_hsm get $2
1330	exist=true
1331	if [ "X$( diff $1 $2 )" = X ] ; then
1332	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
1333	skip=true
1334	else
1335	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
1336	skip=false
1337	fi
1338	fi
1339	#
1340	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
1341	IGCM_sys_Chmod u+w $2
1342	fi
1343
1344	if [ X${skip} = Xfalse ] ; then
1345	i=0
1346	while [ $i -lt $NB_ESSAI ] ; do
1347	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
1348	# USUAL WAY
1349	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1350	status=$?
1351	else
1352	# NOT SO USUAL WAY
1353	\mv $1 $2 > /tmp/out_command.$$ 2>&1
1354	status=$?
1355	fi
1356	if [ ${status} -gt 0 ]; then
1357	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
1358	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
1359	[ -f ${1} ] && ls -l ${1}
1360	[ -f ${2} ] && ls -l ${2}
1361	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1362	sleep $DELAI
1363	else
1364	break
1365	fi
1366	(( i = i + 1 ))
1367	done
1368	fi
1369
1370	# #RSYNC WITH NETWORK SSH CALL
1371	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
1372	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
1373
1374	# #RSYNC WITH NFS USE
1375	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
1376	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
1377
1378	# status=$?
1379	# IGCM_sys_Rsync_out $status
1380
1381	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1382	# (( status=status+$? ))
1383
1384	if [ ${status} -gt 0 ] ; then
1385	echo "IGCM_sys_Put_Out : error."
1386	[ -f ${1} ] && ls -l ${1}
1387	[ -f ${2} ] && ls -l ${2}
1388	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1389	cat /tmp/out_command.$$
1390	IGCM_debug_Exit "IGCM_sys_Put_Out"
1391	else
1392
1393	if [ X${JobType} = XRUN ] ; then
1394	if [ X${3} = X ] ; then
1395	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1396	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1397	fi
1398	fi
1399
1400	\rm /tmp/out_command.$$
1401	fi
1402	fi
1403	IGCM_debug_PopStack "IGCM_sys_Put_Out"
1404	return 0
1405	}
1406
1407	#D-#==================================================
1408	#D-function IGCM_sys_PutBuffer_Out
1409	#D-* Purpose: Copy a file on ${SCRATCHDIR} after having chmod it in readonly
1410	#D-* Examples:
1411	#D-
1412	function IGCM_sys_PutBuffer_Out {
1413	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Out" $@
1414	if ( $DEBUG_sys ) ; then
1415	echo "IGCM_sys_PutBuffer_Out :" $@
1416	fi
1417
1418	typeset NB_ESSAI DELAI status i exist skip
1419
1420	# number of tentative
1421	NB_ESSAI=3
1422	# time delay between tentative
1423	DELAI=2
1424
1425	if [ $DRYRUN = 0 ]; then
1426	if [ ! -f ${1} ] ; then
1427	echo "WARNING : IGCM_sys_PutBuffer_Out ${1} DOES NOT EXIST ."
1428	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
1429	return 1
1430	fi
1431	#
1432	IGCM_sys_Mkdir $( dirname $2 )
1433	#
1434
1435	exist=false
1436	skip=false
1437	if [ -f $2 ] ; then
1438	IGCM_debug_Print 1 "$2 already exist"
1439	exist=true
1440	if [ "X$( diff $1 $2 )" = X ] ; then
1441	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
1442	status=0
1443	skip=true
1444	else
1445	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
1446	skip=false
1447	fi
1448	fi
1449	#
1450	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
1451	IGCM_sys_Chmod u+w $2
1452	fi
1453
1454	if [ X${skip} = Xfalse ] ; then
1455	i=0
1456	while [ $i -lt $NB_ESSAI ] ; do
1457	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
1458	# USUAL WAY
1459	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1460	status=$?
1461	else
1462	# NOT SO USUAL WAY
1463	\mv $1 $2 > /tmp/out_command.$$ 2>&1
1464	status=$?
1465	fi
1466	if [ ${status} -gt 0 ]; then
1467	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
1468	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : sleep ${DELAI} seconds and try again."
1469	[ -f ${2} ] && ls -l ${2}
1470	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1471	sleep $DELAI
1472	else
1473	break
1474	fi
1475	(( i = i + 1 ))
1476	done
1477	fi
1478
1479	if [ ${status} -gt 0 ] ; then
1480	echo "IGCM_sys_PutBuffer_Out : error."
1481	[ -f ${2} ] && ls -l ${2}
1482	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1483	cat /tmp/out_command.$$
1484	IGCM_debug_Exit "IGCM_sys_PutBuffer_Out"
1485	else
1486
1487	if [ X${JobType} = XRUN ] ; then
1488	if [ X${3} = X ] ; then
1489	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1490	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1491	fi
1492	fi
1493
1494	\rm /tmp/out_command.$$
1495	fi
1496	fi
1497	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
1498	return 0
1499	}
1500
1501	#D-#==================================================
1502	#D-function IGCM_sys_Get
1503	#D-* Purpose: Get a file from ${ARCHIVE}
1504	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
1505	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
1506	function IGCM_sys_Get {
1507	IGCM_debug_PushStack "IGCM_sys_Get" $@
1508
1509	typeset DEST dm_liste target file_work
1510	typeset NB_ESSAI DELAI status i
1511
1512	if ( $DEBUG_sys ) ; then
1513	echo "IGCM_sys_Get :" $@
1514	fi
1515
1516	# number of tentative
1517	NB_ESSAI=3
1518	# time delay between tentative
1519	DELAI=2
1520
1521	if [ $DRYRUN -le 2 ]; then
1522	if [ X${1} = X'/l' ] ; then
1523	eval set +A dm_liste \${${2}}
1524	else
1525	eval set +A dm_liste ${1}
1526	fi
1527	eval DEST=\${${#}}
1528
1529	#=====================================================
1530	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
1531	#=====================================================
1532
1533	# Is it an R_OUT file (not R_IN) ?
1534	#echo ${dm_liste[0]} \| grep "${R_OUT}" > /dev/null 2>&1
1535	#if [ $? -eq 0 ] ; then
1536	# # Yes ? then we try to get it in SCRATCHDIR
1537	# set +A file_work $( echo ${dm_liste[*]} \| sed -e "s\|${R_OUT}\|${R_BUF}\|g" )
1538	# if [ -f ${file_work[0]} ] ; then
1539	# IGCM_sys_Cp ${file_work[*]} ${DEST}
1540	# IGCM_debug_PopStack "IGCM_sys_Get"
1541	# return
1542	# fi
1543	#fi
1544
1545	ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1546	status=$?
1547	if [ ${status} -gt 0 ] ; then
1548	echo "WARNING IGCM_sys_Get : error code ${status}"
1549	cat /tmp/out_command.$$
1550	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
1551	fi
1552
1553	#if [ ${status} -gt 0 ] ; then
1554	# if [ ! "X$( grep "Lost dmusrcmd connection" /tmp/out_command.$$ )" = "X" ] ; then
1555	# cat /tmp/out_command.$$
1556	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
1557	# sleep 30
1558	# echo "We try another time"
1559	## dmget ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1560	# ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1561	# status=$?
1562	# if [ ${status} -gt 0 ] ; then
1563	# echo "ERROR IGCM_sys_Get : again demigration error :"
1564	# cat /tmp/out_command.$$
1565	# IGCM_debug_Exit "IGCM_sys_Get"
1566	# fi
1567	# else
1568	# echo "ERROR IGCM_sys_Get : demigration error :"
1569	# cat /tmp/out_command.$$
1570	# IGCM_debug_Exit "IGCM_sys_Get"
1571	# fi
1572	#fi
1573
1574	# #RSYNC WITH NETWORK SSH CALL
1575	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
1576	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
1577
1578	# #RSYNC WITH NFS USE
1579	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
1580	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
1581
1582	# status=$?
1583	# IGCM_sys_Rsync_out $status
1584
1585	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1586	# (( status=status+$? ))
1587
1588	#USUAL WAY
1589	if [ X${1} = X'/l' ] ; then
1590	for target in ${dm_liste[*]} ; do
1591	local_file=$( basename ${target} )
1592	# test if the target file is present before the loop
1593	IGCM_sys_TestFileArchive ${target}
1594	status=$?
1595	if [ ${status} -gt 0 ] ; then
1596	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
1597	IGCM_debug_Exit "IGCM_sys_Get"
1598	else
1599	i=0
1600	while [ $i -lt $NB_ESSAI ] ; do
1601	#if [ X${DoLink} = Xtrue ] ; then
1602	# \ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1603	# status=$?
1604	# else
1605	# \cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1606	# status=$?
1607	#fi
1608	\ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1609	status=$?
1610	if [ ${status} -gt 0 ]; then
1611	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
1612	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
1613	sleep $DELAI
1614	else
1615	break
1616	fi
1617	(( i = i + 1 ))
1618	done
1619	if [ ${status} -gt 0 ] ; then
1620	echo "IGCM_sys_Get : error"
1621	cat /tmp/out_command.$$
1622	\rm /tmp/out_command.$$
1623	IGCM_debug_Exit "IGCM_sys_Get"
1624	else
1625	\rm /tmp/out_command.$$
1626	fi
1627	fi
1628	done
1629	else
1630	i=0
1631	while [ $i -lt $NB_ESSAI ] ; do
1632	\cp ${dm_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
1633	status=$?
1634	if [ ${status} -gt 0 ]; then
1635	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
1636	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
1637	sleep $DELAI
1638	else
1639	break
1640	fi
1641	(( i = i + 1 ))
1642	done
1643	if [ ${status} -gt 0 ] ; then
1644	echo "IGCM_sys_Get : error"
1645	cat /tmp/out_command.$$
1646	\rm /tmp/out_command.$$
1647	IGCM_debug_Exit "IGCM_sys_Get"
1648	else
1649	\rm /tmp/out_command.$$
1650	fi
1651	fi
1652	fi
1653	IGCM_debug_PopStack "IGCM_sys_Get"
1654	}
1655
1656	#D-#==================================================
1657	#D-function IGCM_sys_GetBuffer
1658	#D-* Purpose: Get a file from ${SCRATCHDIR}
1659	#D-* Examples: IGCM_sys_GetBuffer myfile /destpath/myfile_with_PREFIX
1660	#D- IGCM_sys_GetBuffer /l Array_contain_myfiles /destpath/
1661	function IGCM_sys_GetBuffer {
1662	IGCM_debug_PushStack "IGCM_sys_GetBuffer" $@
1663
1664	typeset DEST buf_liste target file_work
1665	typeset NB_ESSAI DELAI status i
1666
1667	if ( $DEBUG_sys ) ; then
1668	echo "IGCM_sys_GetBuffer :" $@
1669	fi
1670
1671	# number of tentative
1672	NB_ESSAI=3
1673	# time delay between tentative
1674	DELAI=2
1675
1676	if [ $DRYRUN -le 2 ]; then
1677	if [ X${1} = X'/l' ] ; then
1678	eval set +A buf_liste \${${2}}
1679	else
1680	eval set +A buf_liste ${1}
1681	fi
1682	eval DEST=\${${#}}
1683
1684	#USUAL WAY
1685	if [ X${1} = X'/l' ] ; then
1686	for target in ${buf_liste[*]} ; do
1687	local_file=$( basename ${target} )
1688	i=0
1689	while [ $i -lt $NB_ESSAI ] ; do
1690	\cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1691	status=$?
1692	if [ ${status} -gt 0 ]; then
1693	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
1694	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
1695	sleep $DELAI
1696	else
1697	break
1698	fi
1699	(( i = i + 1 ))
1700	done
1701	if [ ${status} -gt 0 ] ; then
1702	echo "IGCM_sys_Get : error"
1703	cat /tmp/out_command.$$
1704	\rm /tmp/out_command.$$
1705	IGCM_debug_Exit "IGCM_sys_GetBuffer"
1706	else
1707	\rm /tmp/out_command.$$
1708	fi
1709	done
1710	else
1711	i=0
1712	while [ $i -lt $NB_ESSAI ] ; do
1713	\cp ${buf_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
1714	status=$?
1715	if [ ${status} -gt 0 ]; then
1716	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
1717	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
1718	sleep $DELAI
1719	else
1720	break
1721	fi
1722	(( i = i + 1 ))
1723	done
1724	if [ ${status} -gt 0 ] ; then
1725	echo "IGCM_sys_Get : error"
1726	cat /tmp/out_command.$$
1727	\rm /tmp/out_command.$$
1728	IGCM_debug_Exit "IGCM_sys_GetBuffer"
1729	else
1730	\rm /tmp/out_command.$$
1731	fi
1732	fi
1733	fi
1734	IGCM_debug_PopStack "IGCM_sys_GetBuffer"
1735	}
1736
1737	#D-#==================================================
1738	#D-function IGCM_sys_GetDate_FichWork
1739	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem WORK
1740	#D-* Examples:
1741	#D-
1742	function IGCM_sys_GetDate_FichWork {
1743	IGCM_debug_PushStack "IGCM_sys_FichWork" $@
1744	if ( $DEBUG_sys ) ; then
1745	echo "IGCM_sys_GetDate_FichWork :" $@
1746	fi
1747
1748	if [ $# -ge 3 ] ; then
1749	mode=$3
1750	TimeStyle=$4
1751	else
1752	mode="default"
1753	TimeStyle="%Y%m%d%H%M%S"
1754	fi
1755
1756	typeset dateF
1757	set +A dateF -- $( ls -l --full-time --time-style=+"${TimeStyle}" ${1} )
1758
1759	case $mode in
1760	"default")
1761	eval ${2}=${dateF[5]}
1762	;;
1763	"SplitFields")
1764	eval ${2}="${dateF[5]}\ ${dateF[6]}"
1765	;;
1766	esac
1767
1768
1769	# donne la date filesys d'un fichier sur la machine work
1770	IGCM_debug_PopStack "IGCM_sys_FichWork"
1771	}
1772
1773	#D-#==================================================
1774	#D-function IGCM_sys_GetDate_FichArchive
1775	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem ARCHIVE
1776	#D-* Examples:
1777	#D-
1778	function IGCM_sys_GetDate_FichArchive {
1779	IGCM_debug_PushStack "IGCM_sys_FichArchive" $@
1780	if ( $DEBUG_sys ) ; then
1781	echo "IGCM_sys_GetDate_FichArchive :" $@
1782	fi
1783	typeset dateF
1784	set +A dateF -- $( ls -l --full-time --time-style='+%Y%m%d%H%M%S' ${1} )
1785	eval ${2}=${dateF[5]}
1786
1787	IGCM_debug_PopStack "IGCM_sys_FichArchive"
1788	}
1789
1790	#D-#==================================================
1791	#D-function IGCM_sys_GetDate_Monitoring
1792	#D-* Purpose: get the last year for which the monitoring has been computed
1793	#D-* Examples:
1794	#D-
1795	function IGCM_sys_GetDate_Monitoring {
1796	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
1797	if ( $DEBUG_sys ) ; then
1798	echo "IGCM_sys_GetDate_Monitoring :" $@
1799	fi
1800
1801	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
1802
1803	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
1804	}
1805
1806	#D-#==================================================
1807	#D-function IGCM_sys_Dods_Rm
1808	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
1809	#D-* Examples:
1810	#D-
1811	function IGCM_sys_Dods_Rm {
1812	if ( $DEBUG_sys ) ; then
1813	echo "IGCM_sys_Dods_Rm :" $@
1814	fi
1815	typeset status
1816	if [ $DRYRUN = 0 ]; then
1817
1818	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
1819	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
1820	# echo "Nothing has been done."
1821	# return
1822	# fi
1823
1824	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
1825	status=$?
1826
1827	# if [ ${status} -gt 0 ] ; then
1828	# echo "IGCM_sys_Dods_Rm : error."
1829	# cat out_dods_rm
1830	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
1831	# else
1832	# rm out_dods_rm
1833	# fi
1834
1835	fi
1836	return $status
1837	}
1838
1839	#D-#==================================================
1840	#D-function IGCM_sys_Dods_Cp
1841	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
1842	#D-* Examples:
1843	#D-
1844	function IGCM_sys_Dods_Cp {
1845	if ( $DEBUG_sys ) ; then
1846	echo "IGCM_sys_Dods_Cp :" $@
1847	fi
1848	typeset status
1849	if [ $DRYRUN = 0 ]; then
1850
1851	# if [ ! -d ${R_SAVE}/${1} ] ; then
1852	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
1853	# echo "Nothing has been done."
1854	# return
1855	# fi
1856
1857	/ccc/cont003/home/dsm/p86ipsl/bin/dods_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
1858	status=$?
1859
1860	# if [ ${status} -gt 0 ] ; then
1861	# echo "IGCM_sys_Dods_Cp : error."
1862	# cat out_dods_cp
1863	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1864	# else
1865	# rm out_dods_cp
1866	# fi
1867
1868	fi
1869	return $status
1870	}
1871
1872	#D-#==================================================
1873	#D-function IGCM_sys_Put_Dods
1874	#D-* Purpose: Put $(ARCHIVE) files on DODS internet protocole.
1875	#D-* Examples:
1876	#D-
1877	function IGCM_sys_Put_Dods {
1878	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1879	if ( $DEBUG_sys ) ; then
1880	echo "IGCM_sys_Put_Dods :" $@
1881	fi
1882	#set -vx
1883	typeset status
1884	if [ $DRYRUN = 0 ]; then
1885	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1886	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1887	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1888	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1889	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1890	return
1891	fi
1892
1893	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1894	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1895	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1896	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1897	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1898	return
1899	fi
1900	#
1901	if [ -d ${R_SAVE}/${1} ] ; then
1902	cd ${R_SAVE}
1903	elif [ -d ${R_FIGR}/${1} ] ; then
1904	cd ${R_FIGR}
1905	fi
1906
1907	IGCM_sys_Dods_Rm ${1}
1908	IGCM_sys_Dods_Cp ${1}
1909	status=0
1910
1911	if [ ${status} -gt 0 ] ; then
1912	echo "IGCM_sys_Put_Dods : error."
1913	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1914	fi
1915	fi
1916	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1917	}
1918
1919	##############################################################
1920	# REBUILD OPERATOR
1921
1922	#D-#==================================================
1923	#D-function IGCM_sys_rebuild
1924	#D-* Purpose: rebuild parallel files
1925	#D-* Examples:
1926	#D-
1927	function IGCM_sys_rebuild {
1928	IGCM_debug_PushStack "IGCM_sys_rebuild" $@
1929	if ( $DEBUG_sys ) ; then
1930	echo "IGCM_sys_rebuild :" $@
1931	fi
1932
1933	typeset NB_ESSAI DELAI status i firstArg
1934	# number of tentative
1935	NB_ESSAI=3
1936	# time delay between tentative
1937	DELAI=2
1938
1939	i=0
1940	while [ $i -lt $NB_ESSAI ] ; do
1941	~p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -f -o $@ > /tmp/out_command.$$ 2>&1
1942	status=$?
1943	if [ ${status} -gt 0 ] ; then
1944	IGCM_debug_Print 2 "IGCM_sys_rebuild : error code ${status}"
1945	cat /tmp/out_command.$$
1946	\rm /tmp/out_command.$$
1947	IGCM_debug_Print 2 "IGCM_sys_rebuild : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
1948	firstArg=${1}
1949	\rm ${firstArg}
1950	sleep $DELAI
1951	else
1952	\rm /tmp/out_command.$$
1953	break
1954	fi
1955	(( i = i + 1 ))
1956	done
1957
1958	if [ ${status} -gt 0 ] ; then
1959	echo "IGCM_sys_rebuild : rebuild error code is ${status}"
1960	IGCM_debug_Exit "rebuild"
1961	fi
1962
1963	IGCM_debug_PopStack "IGCM_sys_rebuild"
1964	}
1965
1966	#D-#==================================================
1967	#D-function IGCM_sys_rebuild_station
1968	#D-* Purpose: rebuild parallel files describing station
1969	#D-* Examples:
1970	#D-
1971	function IGCM_sys_rebuild_station {
1972	IGCM_debug_PushStack "IGCM_sys_rebuild_station" $@
1973	typeset i list_opt file_in file_out prefix_invert list_invert
1974	if ( $DEBUG_sys ) ; then
1975	echo "IGCM_sys_rebuild_station :" $@
1976	fi
1977	list_opt=$@
1978
1979	# Invert Axis : t,x -> x,t
1980	# t,pres,x -> x,t,pres
1981	# So that we can concatenate along x
1982	i=0
1983	for file_in in ${list_opt} ; do
1984	(( i = i + 1))
1985	[ ${i} = 1 ] && file_out=${file_in} && continue
1986	# detect time counter and do the job only if present
1987	var_unlim=$(ncdump -h ${file_in} \| grep UNLIMITED \| cut -d ' ' -f 1 \| sed -e 's/^[ \t]//' -e 's/[ \t]$//')
1988	if [ X${var_unlim} = Xtime_counter ] ; then
1989	prefix_invert=$( basename ${file_in} .nc )
1990	IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs ${file_in} ${prefix_invert}_xt.nc
1991	list_invert[${#list_invert[*]}]=${prefix_invert}_xt.nc
1992	fi
1993	done
1994
1995	# Concatenate
1996	IGCM_sys_ncrcat ${list_invert[*]} histstn_xt.nc
1997
1998	# Re-ivert file
1999	IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x histstn_xt.nc ${file_out}
2000
2001	# Station re-ordering is too expansive to be run within libIGCM
2002	# This is due to (ncpdq - nrcat - ncpdq) I/O sequence.
2003	# This re-ordering must be done "in memory" by the cmorization process
2004	# Anyway this is the best sequence using (ncpdq - nrcat - ncpdq)
2005	# BEGIN reordering
2006
2007	# Only LMDZ text output contains the exact ordering of the station.
2008	# We isolate this in the code below:
2009	# 0 38 -157.5000000000000 70.98591549295774
2010	# 0 54 27.49999999999999 67.18309859154928
2011	# 0 56 -62.50000000000001 82.39436619718309
2012	# 0 79 12.49999999999999 78.59154929577466
2013	# 0 116 -165.0000000000000 76.05633802816901
2014	# 0 117 130.0000000000000 70.98591549295774
2015	# 0 118 110.0000000000000 87.46478873239437
2016	# 1 40 4.999999999999995 51.97183098591550
2017
2018	# typeset iStation iProc list_opt file_in file_out prefix_invert
2019	# typeset -Z4 j4
2020	# typeset -Z3 j3
2021
2022	# unset list_opt
2023	# set +A list_opt $@
2024
2025	# Filename after rebuild
2026	# file_out=${list_opt[0]}
2027	# Prefix of output files
2028	# prefix_invert=$( basename ${file_out} .nc )
2029	# Number of procs
2030	# num_proc=$( grep -i mpi_size ${PREFIX}_${Exe_Output} \| wc -l )
2031
2032	# iProc=0
2033	# while [ ${iProc} -lt ${num_proc} ] ; do
2034	# Array containing Station as a number
2035	# unset proc_stn
2036	# set +A proc_stn $( grep "iophy_mpi rank ip lon lat $iProc" ${PREFIX}_${Exe_Output} \| sed -e "s/iophy_mpi rank ip lon lat //g" \| gawk ' {print $2}' )
2037	# Number of stations produced by processor proc
2038	# stationLast=${#proc_stn[*]}
2039	# Proc number on 4 digits
2040	# j4=${iProc}
2041	# Init
2042	# iStation=0
2043	# while [ ${iStation} -lt ${stationLast} ] ; do
2044	# Station number on 3 digits
2045	# j3=${proc_stn[${iStation}]}
2046	# Extract station
2047	# Invert Axis : t,x -> x,t
2048	# t,pres,x -> x,t,pres
2049	# So that we can concatenate along x
2050	# IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs -d x,$iStation,$iStation ${prefix_invert}_${j4}.nc ${prefix_invert}_stn_${j3}.nc
2051	# (( iStation = iStation + 1 ))
2052	# done
2053	# (( iProc = iProc + 1 ))
2054	# done
2055
2056	# Concatenate all station along x
2057	# IGCM_sys_ncrcat ${prefix_invert}_stn_???.nc ${prefix_invert}_xt.nc
2058
2059	# Re-invert file
2060	# IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x ${prefix_invert}_xt.nc ${file_out}
2061
2062	# END reordering
2063
2064	IGCM_debug_PopStack "IGCM_sys_rebuild_station"
2065	}
2066
2067	############################################################
2068	# Activate Running Environnment Variables
2069
2070	#D-#==================================================
2071	#D-function IGCM_sys_desactiv_variables
2072	#D-* Purpose: set environement variables prior to execution
2073	#D-* Examples:
2074	#D-
2075	function IGCM_sys_activ_variables {
2076	IGCM_debug_PushStack "IGCM_sys_activ_variables"
2077	if ( $DEBUG_sys ) ; then
2078	echo "IGCM_sys_activ_variables"
2079	fi
2080
2081	# --------------------------------------------------------------------
2082	#D- MPI specifications
2083	# --------------------------------------------------------------------
2084
2085	# --------------------------------------------------------------------
2086	#D- Other specifications
2087	# --------------------------------------------------------------------
2088
2089	ulimit -s unlimited
2090
2091	IGCM_debug_PopStack "IGCM_sys_activ_variables"
2092	}
2093
2094	############################################################
2095	# Desactivate Running Environnment Variables
2096
2097	#D-#==================================================
2098	#D-function IGCM_sys_desactiv_variables
2099	#D-* Purpose: unset environement variables after execution
2100	#D-* Examples:
2101	#D-
2102	function IGCM_sys_desactiv_variables {
2103	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
2104	if ( $DEBUG_sys ) ; then
2105	echo "IGCM_sys_desactiv_variables"
2106	fi
2107	# --------------------------------------------------------------------
2108	#D- MPI specifications
2109	# --------------------------------------------------------------------
2110
2111	# --------------------------------------------------------------------
2112	#D- Other specifications
2113	# --------------------------------------------------------------------
2114
2115	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
2116	}
2117
2118	############################################################
2119	# Build MPI/OMP scripts run file (dummy function)
2120
2121	#D-#==================================================
2122	#D-function IGCM_sys_build_run_file
2123	#D-* Purpose: build run file (deprecated)
2124	#D-* Examples:
2125	#D-
2126	function IGCM_sys_build_run_file {
2127
2128	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
2129
2130	}
2131
2132	############################################################
2133	# Build MPI/OMP scripts
2134
2135	#D-#==================================================
2136	#D-function IGCM_sys_build_execution_scripts
2137	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
2138	#D-* Examples:
2139	#D-
2140	function IGCM_sys_build_execution_scripts
2141	{
2142	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
2143	if ( $DEBUG_sys ) ; then
2144	echo "IGCM_sys_build_execution_scripts " $@
2145	fi
2146
2147	EXECUTION=${HOST_MPIRUN_COMMAND}
2148
2149	# MPMD mode
2150	if ( ${OK_PARA_MPMD} ) ; then
2151
2152	# Only MPI (MPMD)
2153	if ( ! ${OK_PARA_OMP} ) ; then
2154
2155	if [ -f run_file ] ; then
2156	IGCM_sys_Rm -f run_file
2157	fi
2158	touch run_file
2159
2160	# Build run_file
2161
2162	# First loop on the components for the coupler ie oasis (only if oasis3)
2163	# the coupler ie oasis3 must be the first one
2164	for comp in ${config_ListOfComponents[*]} ; do
2165
2166	eval ExeNameIn=\${config_Executable_${comp}[0]}
2167	eval ExeNameOut=\${config_Executable_${comp}[1]}
2168
2169	# for CPL component only
2170	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
2171
2172	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2173	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2174	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
2175	fi
2176	done
2177
2178	# Then second loop on the components
2179	for comp in ${config_ListOfComponents[*]} ; do
2180
2181	eval ExeNameIn=\${config_Executable_${comp}[0]}
2182	eval ExeNameOut=\${config_Executable_${comp}[1]}
2183
2184	# Only if we really have an executable for the component and not the coupler ie oasis:
2185	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
2186
2187	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2188	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2189	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
2190	fi
2191	done
2192
2193	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
2194
2195	IGCM_sys_Chmod u+x run_file
2196	if ( $DEBUG_sys ) ; then
2197	echo "run_file contains : "
2198	cat run_file
2199	fi
2200
2201	# MPI-OpenMP (MPMD)
2202	else
2203
2204	# Use of mpirun instead of ccc_mprun
2205	EXECUTION="time mpirun"
2206
2207	# Hosts treatment
2208	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
2209
2210	i=0
2211	rm -f hosts rankfile
2212	IGCM_debug_Print 1 "sys Curie, Hosts available :"
2213	for nodes in `cat hosts.tmp`
2214	do
2215	host[$i]=$nodes
2216	echo "${host[$i]}" >> hosts
2217	IGCM_debug_Print 1 ${host[$i]}
2218	i=$((i+1))
2219	done
2220	rm -f hosts.tmp
2221
2222	listnodes=${host[*]}
2223
2224	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
2225
2226	# Initialisation
2227	rank=0
2228	current_core=0
2229	core_per_node=16
2230	init_exec=n
2231
2232	# Loop on the components
2233	for comp in ${config_ListOfComponents[*]} ; do
2234
2235	eval ExeNameIn=\${config_Executable_${comp}[0]}
2236	eval ExeNameOut=\${config_Executable_${comp}[1]}
2237
2238	# Not possible if oasis has an executable (i.e old version of oasis3)
2239	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
2240	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
2241	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
2242	IGCM_debug_Verif_Exit
2243	fi
2244
2245	# Only if we really have an executable for the component :
2246	if [ "X${ExeNameOut}" != X\"\" ] ; then
2247
2248	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2249	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2250
2251	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
2252	echo "" >> script_${ExeNameOut}.ksh
2253	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
2254
2255	# Check if the number of threads is correct
2256	case ${comp_proc_omp_loc} in
2257	2\|4\|8\|16)
2258	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
2259	;;
2260	*)
2261	IGCM_debug_Exit "ERROR with OMP parameters !"
2262	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
2263	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
2264	IGCM_debug_Verif_Exit
2265	;;
2266	esac
2267	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
2268	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
2269	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
2270	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
2271	fi
2272	#echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
2273	#echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
2274	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2275	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
2276
2277	if [ ${init_exec} = y ] ; then
2278	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2279	else
2280	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2281	init_exec=y
2282	fi
2283
2284	# Build rankfile : method used to assign cores and nodes for the MPI process
2285	# Ex :
2286	#rank 0=curie5296 slot=0,1,2,3
2287	#rank 1=curie5296 slot=4,5,6,7
2288	# Example of final command :
2289	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
2290	# with script_lmdz.x.ksh :
2291	# #!/bin/ksh
2292	#export KMP_STACKSIZE=3g
2293	#export KMP_LIBRARY=turnaround
2294	#export MKL_SERIAL=YES
2295	#OMP_NUM_THREADS=4
2296	#./lmdz.x
2297	#
2298
2299	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
2300	(( index_host = current_core / core_per_node ))
2301	host_value=${host[${index_host}]}
2302	(( slot = current_core % core_per_node ))
2303	virg=","
2304	string_final=""
2305	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
2306	string=$index$virg
2307	string_final=$string_final$string
2308	done
2309	string_final=$( echo $string_final \| sed "s/.$//" )
2310	echo "rank $rank=$host_value slot=$string_final" >> rankfile
2311	(( rank = rank + 1 ))
2312	(( current_core = current_core + comp_proc_omp_loc ))
2313	done
2314	fi
2315
2316	done
2317	fi
2318
2319	# Only one executable (SPMD mode).
2320	else
2321
2322	for comp in ${config_ListOfComponents[*]} ; do
2323
2324	# Only if we really have an executable for the component :
2325	eval ExeNameOut=\${config_Executable_${comp}[1]}
2326	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
2327
2328	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
2329	echo "" >> script_${ExeNameOut}.ksh
2330	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
2331
2332	if ( ${OK_PARA_OMP} ) ; then
2333	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2334	echo "" >> script_${ExeNameOut}.ksh
2335	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
2336	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
2337	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
2338	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
2339	fi
2340
2341	if ( ${OK_PARA_MPI} ) ; then
2342	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2343	# Default : ccc_mprun used if nb_proc gt 1
2344	# to have out/err per process on different files
2345	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
2346	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2347	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2348	else
2349	# Default : ccc_mprun is NOT used if nb_proc eq 1
2350	# to have out/err per process on different files
2351	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
2352	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2353	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
2354	fi
2355
2356	IGCM_debug_Print 1 "sys Curie : script_${ExeNameOut}.ksh contains"
2357	cat script_${ExeNameOut}.ksh
2358
2359	fi
2360
2361	done
2362
2363	fi
2364
2365	IGCM_debug_Print 1 "sys Curie : execution command is "
2366	IGCM_debug_Print 1 "$EXECUTION"
2367
2368	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
2369	}
2370
2371	#D-#==================================================
2372	#D-function IGCM_sys_check_path
2373	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
2374	#D-* do not point to an important use directory. Stop immediately in that case.
2375	#D-* Examples:
2376	#D-
2377	function IGCM_sys_check_path {
2378	IGCM_debug_PushStack "IGCM_sys_check_path"
2379	if ( $DEBUG_sys ) ; then
2380	echo "IGCM_sys_check_path"
2381	fi
2382
2383	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
2384	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
2385	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
2386	IGCM_debug_Exit "This will stop the job"
2387	fi
2388	IGCM_debug_PopStack "IGCM_sys_check_path"
2389	}
2390
2391	#D-#==================================================
2392	#D-function IGCM_sys_check_quota
2393	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
2394	#D-* Examples:
2395	#D-
2396	function IGCM_sys_check_quota {
2397	IGCM_debug_PushStack "IGCM_sys_check_quota"
2398	if ( $DEBUG_sys ) ; then
2399	echo "IGCM_sys_check_quota"
2400	fi
2401	# Limit of quota (in %)
2402	limit_quota=90
2403
2404	# Check of the volume
2405	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
2406	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
2407
2408	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
2409
2410	unit_avail=${volume_avail: -1}
2411	unit_quota=${volume_quota: -1}
2412
2413	if [ "${unit_quota}" = "*" ] ; then
2414	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
2415	IGCM_debug_Print 1 "More than 100% of your quota is used"
2416	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2417	IGCM_debug_Print 1 "You must have more than 10% available to run"
2418	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2419	IGCM_debug_Verif_Exit
2420	fi
2421
2422	temp_avail=${volume_avail%%${volume_avail: -1}*}
2423	temp_quota=${volume_quota%%${volume_quota: -1}*}
2424
2425	if [ ! ${unit_avail} = ${unit_quota} ] ; then
2426
2427	# Convertion
2428	if [ ${volume_avail: -1} = "T" ] ; then
2429	(( temp_avail = temp_avail * 1000000000000 ))
2430	elif [ ${volume_avail: -1} = "G" ] ; then
2431	(( temp_avail = temp_avail * 1000000000 ))
2432	elif [ ${volume_avail: -1} = "M" ] ; then
2433	(( temp_avail = temp_avail * 1000000 ))
2434	elif [ ${volume_avail: -1} = "k" ] ; then
2435	(( temp_avail = temp_avail * 1000 ))
2436	else
2437	(( temp_avail = volume_avail ))
2438	fi
2439	if [ ${volume_quota: -1} = "T" ] ; then
2440	(( temp_quota = temp_quota * 1000000000000 ))
2441	elif [ ${volume_quota: -1} = "G" ] ; then
2442	(( temp_quota = temp_quota * 1000000000 ))
2443	elif [ ${volume_quota: -1} = "M" ] ; then
2444	(( temp_quota = temp_quota * 1000000 ))
2445	elif [ ${volume_quota: -1} = "k" ] ; then
2446	(( temp_quota = temp_quota * 1000 ))
2447	else
2448	(( temp_quota = volume_quota ))
2449	fi
2450	fi
2451
2452	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
2453	# echo "volume ratio is " $quota_volume
2454
2455	if [ ${quota_volume} -ge ${limit_quota} ] ; then
2456	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
2457	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
2458	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2459	IGCM_debug_Print 1 "You must have more than 10% available to run"
2460	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2461	IGCM_debug_Verif_Exit
2462	fi
2463
2464	fi
2465
2466	# Check of the number of inodes
2467
2468	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
2469	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
2470
2471	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
2472
2473	unit_avail=${inode_avail: -1}
2474	unit_quota=${inode_quota: -1}
2475
2476	if [ "${unit_quota}" = "*" ] ; then
2477	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
2478	IGCM_debug_Print 1 "More than 100% of your quota is used"
2479	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2480	IGCM_debug_Print 1 "You must have more than 10% available to run"
2481	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2482	IGCM_debug_Verif_Exit
2483	fi
2484
2485	temp_avail=${inode_avail%%${inode_avail: -1}*}
2486	temp_quota=${inode_quota%%${inode_quota: -1}*}
2487
2488	if [ ! ${unit_avail} = ${unit_quota} ] ; then
2489
2490	# Convertion
2491	if [ ${inode_avail: -1} = "T" ] ; then
2492	(( temp_avail = temp_avail * 1000000000000 ))
2493	elif [ ${inode_avail: -1} = "G" ] ; then
2494	(( temp_avail = temp_avail * 1000000000 ))
2495	elif [ ${inode_avail: -1} = "M" ] ; then
2496	(( temp_avail = temp_avail * 1000000 ))
2497	elif [ ${inode_avail: -1} = "k" ] ; then
2498	(( temp_avail = temp_avail * 1000 ))
2499	else
2500	(( temp_avail = inode_avail ))
2501	fi
2502
2503	if [ ${inode_quota: -1} = "T" ] ; then
2504	(( temp_quota = temp_quota * 1000000000000 ))
2505	elif [ ${inode_quota: -1} = "G" ] ; then
2506	(( temp_quota = temp_quota * 1000000000 ))
2507	elif [ ${inode_quota: -1} = "M" ] ; then
2508	(( temp_quota = temp_quota * 1000000 ))
2509	elif [ ${inode_quota: -1} = "k" ] ; then
2510	(( temp_quota = temp_quota * 1000 ))
2511	else
2512	(( temp_quota = inode_quota ))
2513	fi
2514	fi
2515	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
2516	# echo "inode ratio is " $quota_inode
2517
2518	if [ ${quota_inode} -ge ${limit_quota} ] ; then
2519	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
2520	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
2521	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2522	IGCM_debug_Print 1 "You must have more than 10% available to run"
2523	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2524	IGCM_debug_Verif_Exit
2525	fi
2526	fi
2527	IGCM_debug_PopStack "IGCM_sys_check_quota"
2528	}
2529
2530	#D-#==================================================
2531	#D-function IGCM_sys_GetJobID
2532	#D-* Purpose: Check if job_name is currently
2533	#D- running or in queue
2534	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
2535	#D-
2536	function IGCM_sys_GetJobID {
2537	IGCM_debug_PushStack "IGCM_sys_GetJobID"
2538	if ( $DEBUG_sys ) ; then
2539	echo "IGCM_sys_GetJobID"
2540	fi
2541
2542	# With -f option, the full job name is given in the last column
2543	ID=$( ccc_mstat -f -u $2 \| \
2544	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
2545
2546	eval ${3}=${ID}
2547
2548	IGCM_debug_PopStack "IGCM_sys_GetJobID"
2549	}
2550
2551	#D-#==================================================
2552	#D-function IGCM_sys_CountJobInQueue
2553	#D-* Purpose: Check if job_name is currently
2554	#D- running or in queue
2555	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
2556	#D-
2557	function IGCM_sys_CountJobInQueue {
2558	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
2559	if ( $DEBUG_sys ) ; then
2560	echo "IGCM_sys_CountJobInQueue"
2561	fi
2562
2563	#NbRun=$( ccc_mstat -f \| grep -c ${JobName} )
2564
2565	# With -f option, the full job name is given in the last column
2566	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
2567
2568	eval ${2}=${NbRun}
2569
2570	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
2571	}
2572
2573	#D-#==================================================
2574	#D-function IGCM_sys_ListJobInQueue
2575	#D-* Purpose: Check if job_name is currently
2576	#D- running or in queue
2577	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
2578	#D-
2579	function IGCM_sys_ListJobInQueue {
2580	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
2581	if ( $DEBUG_sys ) ; then
2582	echo "IGCM_sys_ListJobInQueue"
2583	fi
2584
2585	# With -f option, the full job name is given in the last column
2586	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
2587	'( $2 == User && \
2588	$NF != /TS/ && \
2589	$NF !~ /PACK/ && \
2590	$NF !~ /REBUILD/ && \
2591	$NF !~ /pack/ ) \
2592	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
2593
2594	eval set -A ${2} ${JobList[*]}
2595
2596	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
2597	}
2598
2599	##############################################################
2600	# NCO OPERATOR
2601
2602	#D-#==================================================
2603	#D-function IGCM_sys_ncap2
2604	#D-* Purpose: encapsulate ncap2 call so as to manage error code and retry
2605	#D-* Examples:
2606	#D-
2607	function IGCM_sys_ncap2 {
2608	IGCM_debug_PushStack "IGCM_sys_ncap2" $@
2609	if ( $DEBUG_sys ) ; then
2610	echo "IGCM_sys_ncap2 :" $@
2611	fi
2612
2613	typeset NB_ESSAI DELAI status i
2614	# number of tentative
2615	NB_ESSAI=3
2616	# time delay between tentative
2617	DELAI=2
2618
2619	i=0
2620	while [ $i -lt $NB_ESSAI ] ; do
2621	ncap2 -C "$@" > /tmp/out_command.$$ 2>&1
2622	status=$?
2623	if [ ${status} -gt 0 ] ; then
2624	IGCM_debug_Print 2 "IGCM_sys_ncap2 : error code ${status}"
2625	cat /tmp/out_command.$$
2626	\rm /tmp/out_command.$$
2627	IGCM_debug_Print 2 "IGCM_sys_ncap2 : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2628	sleep $DELAI
2629	else
2630	\rm /tmp/out_command.$$
2631	break
2632	fi
2633	(( i = i + 1 ))
2634	done
2635
2636	if [ ${status} -gt 0 ] ; then
2637	echo "IGCM_sys_ncap2 : ncap2 error"
2638	IGCM_debug_Exit "ncap2"
2639	fi
2640
2641	IGCM_debug_PopStack "IGCM_sys_ncap2"
2642	}
2643
2644	#D-#==================================================
2645	#D-function IGCM_sys_ncatted
2646	#D-* Purpose: encapsulate ncatted call so as to manage error code and retry
2647	#D-* Examples:
2648	#D-
2649	function IGCM_sys_ncatted {
2650	IGCM_debug_PushStack "IGCM_sys_ncatted" $@
2651	if ( $DEBUG_sys ) ; then
2652	echo "IGCM_sys_ncatted :" $@
2653	fi
2654
2655	typeset NB_ESSAI DELAI status i
2656	# number of tentative
2657	NB_ESSAI=3
2658	# time delay between tentative
2659	DELAI=2
2660
2661	i=0
2662	while [ $i -lt $NB_ESSAI ] ; do
2663	ncatted "$@" > /tmp/out_command.$$ 2>&1
2664	status=$?
2665	if [ ${status} -gt 0 ] ; then
2666	IGCM_debug_Print 2 "IGCM_sys_ncatted : error code ${status}"
2667	cat /tmp/out_command.$$
2668	\rm /tmp/out_command.$$
2669	IGCM_debug_Print 2 "IGCM_sys_ncatted : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2670	sleep $DELAI
2671	else
2672	\rm /tmp/out_command.$$
2673	break
2674	fi
2675	(( i = i + 1 ))
2676	done
2677
2678	if [ ${status} -gt 0 ] ; then
2679	echo "IGCM_sys_ncatted : ncatted error"
2680	IGCM_debug_Exit "ncatted"
2681	fi
2682
2683	IGCM_debug_PopStack "IGCM_sys_ncatted"
2684	}
2685
2686	#D-#==================================================
2687	#D-function IGCM_sys_ncbo
2688	#D-* Purpose: encapsulate ncbo call so as to manage error code and retry
2689	#D-* Examples:
2690	#D-
2691	function IGCM_sys_ncbo {
2692	IGCM_debug_PushStack "IGCM_sys_ncbo" $@
2693	if ( $DEBUG_sys ) ; then
2694	echo "IGCM_sys_ncbo :" $@
2695	fi
2696
2697	typeset NB_ESSAI DELAI status i
2698	# number of tentative
2699	NB_ESSAI=3
2700	# time delay between tentative
2701	DELAI=2
2702
2703	i=0
2704	while [ $i -lt $NB_ESSAI ] ; do
2705	ncbo -C $@ > /tmp/out_command.$$ 2>&1
2706	status=$?
2707	if [ ${status} -gt 0 ] ; then
2708	IGCM_debug_Print 2 "IGCM_sys_ncbo : error code ${status}"
2709	cat /tmp/out_command.$$
2710	\rm /tmp/out_command.$$
2711	IGCM_debug_Print 2 "IGCM_sys_ncbo : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2712	sleep $DELAI
2713	else
2714	\rm /tmp/out_command.$$
2715	break
2716	fi
2717	(( i = i + 1 ))
2718	done
2719
2720	if [ ${status} -gt 0 ] ; then
2721	echo "IGCM_sys_ncbo : ncbo error"
2722	IGCM_debug_Exit "ncbo"
2723	fi
2724
2725	IGCM_debug_PopStack "IGCM_sys_ncbo"
2726	}
2727
2728	#D-#==================================================
2729	#D-function IGCM_sys_ncdif
2730	#D-* Purpose: encapsulate ncdiff call so as to manage error code and retry
2731	#D-* Examples:
2732	#D-
2733	function IGCM_sys_ncdiff {
2734	IGCM_debug_PushStack "IGCM_sys_ncdiff" $@
2735	if ( $DEBUG_sys ) ; then
2736	echo "IGCM_sys_ncdiff :" $@
2737	fi
2738
2739	typeset NB_ESSAI DELAI status i
2740	# number of tentative
2741	NB_ESSAI=3
2742	# time delay between tentative
2743	DELAI=2
2744
2745	i=0
2746	while [ $i -lt $NB_ESSAI ] ; do
2747	ncdiff -C $@ > /tmp/out_command.$$ 2>&1
2748	status=$?
2749	if [ ${status} -gt 0 ] ; then
2750	IGCM_debug_Print 2 "IGCM_sys_ncdiff : error code ${status}"
2751	cat /tmp/out_command.$$
2752	\rm /tmp/out_command.$$
2753	IGCM_debug_Print 2 "IGCM_sys_ncdiff : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2754	sleep $DELAI
2755	else
2756	\rm /tmp/out_command.$$
2757	break
2758	fi
2759	(( i = i + 1 ))
2760	done
2761
2762	if [ ${status} -gt 0 ] ; then
2763	echo "IGCM_sys_ncdiff : ncdiff error"
2764	IGCM_debug_Exit "ncdiff"
2765	fi
2766
2767	IGCM_debug_PopStack "IGCM_sys_ncdiff"
2768	}
2769
2770	#D-#==================================================
2771	#D-function IGCM_sys_ncea
2772	#D-* Purpose: encapsulate ncea call so as to manage error code and retry
2773	#D-* Examples:
2774	#D-
2775	function IGCM_sys_ncea {
2776	IGCM_debug_PushStack "IGCM_sys_ncea" $@
2777	if ( $DEBUG_sys ) ; then
2778	echo "IGCM_sys_ncea :" $@
2779	fi
2780
2781	typeset NB_ESSAI DELAI status i
2782	# number of tentative
2783	NB_ESSAI=3
2784	# time delay between tentative
2785	DELAI=2
2786
2787	i=0
2788	while [ $i -lt $NB_ESSAI ] ; do
2789	ncea -C $@ > /tmp/out_command.$$ 2>&1
2790	status=$?
2791	if [ ${status} -gt 0 ] ; then
2792	IGCM_debug_Print 2 "IGCM_sys_ncea : error code ${status}"
2793	cat /tmp/out_command.$$
2794	\rm /tmp/out_command.$$
2795	IGCM_debug_Print 2 "IGCM_sys_ncea : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2796	sleep $DELAI
2797	else
2798	\rm /tmp/out_command.$$
2799	break
2800	fi
2801	(( i = i + 1 ))
2802	done
2803
2804	if [ ${status} -gt 0 ] ; then
2805	echo "IGCM_sys_ncea : ncea error"
2806	IGCM_debug_Exit "ncea"
2807	fi
2808
2809	IGCM_debug_PopStack "IGCM_sys_ncea"
2810	}
2811
2812	#D-#==================================================
2813	#D-function IGCM_sys_ncecat
2814	#D-* Purpose: encapsulate ncecat call so as to manage error code and retry
2815	#D-* Examples:
2816	#D-
2817	function IGCM_sys_ncecat {
2818	IGCM_debug_PushStack "IGCM_sys_ncecat" $@
2819	if ( $DEBUG_sys ) ; then
2820	echo "IGCM_sys_ncecat :" $@
2821	fi
2822
2823	typeset NB_ESSAI DELAI status i
2824	# number of tentative
2825	NB_ESSAI=3
2826	# time delay between tentative
2827	DELAI=2
2828
2829	i=0
2830	while [ $i -lt $NB_ESSAI ] ; do
2831	ncecat -C $@ > /tmp/out_command.$$ 2>&1
2832	status=$?
2833	if [ ${status} -gt 0 ] ; then
2834	IGCM_debug_Print 2 "IGCM_sys_ncecat : error code ${status}"
2835	cat /tmp/out_command.$$
2836	\rm /tmp/out_command.$$
2837	IGCM_debug_Print 2 "IGCM_sys_ncecat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2838	sleep $DELAI
2839	else
2840	\rm /tmp/out_command.$$
2841	break
2842	fi
2843	(( i = i + 1 ))
2844	done
2845
2846	if [ ${status} -gt 0 ] ; then
2847	echo "IGCM_sys_ncecat : ncecat error"
2848	IGCM_debug_Exit "ncecat"
2849	fi
2850
2851	IGCM_debug_PopStack "IGCM_sys_ncecat"
2852	}
2853
2854	#D-#==================================================
2855	#D-function IGCM_sys_ncflint
2856	#D-* Purpose: encapsulate ncflint call so as to manage error code and retry
2857	#D-* Examples:
2858	#D-
2859	function IGCM_sys_ncflint {
2860	IGCM_debug_PushStack "IGCM_sys_ncflint" $@
2861	if ( $DEBUG_sys ) ; then
2862	echo "IGCM_sys_ncflint :" $@
2863	fi
2864
2865	typeset NB_ESSAI DELAI status i
2866	# number of tentative
2867	NB_ESSAI=3
2868	# time delay between tentative
2869	DELAI=2
2870
2871	i=0
2872	while [ $i -lt $NB_ESSAI ] ; do
2873	ncflint -C $@ > /tmp/out_command.$$ 2>&1
2874	status=$?
2875	if [ ${status} -gt 0 ] ; then
2876	IGCM_debug_Print 2 "IGCM_sys_ncflint : error code ${status}"
2877	cat /tmp/out_command.$$
2878	\rm /tmp/out_command.$$
2879	IGCM_debug_Print 2 "IGCM_sys_ncflint : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2880	sleep $DELAI
2881	else
2882	\rm /tmp/out_command.$$
2883	break
2884	fi
2885	(( i = i + 1 ))
2886	done
2887
2888	if [ ${status} -gt 0 ] ; then
2889	echo "IGCM_sys_ncflint : ncflint error"
2890	IGCM_debug_Exit "ncflint"
2891	fi
2892
2893	IGCM_debug_PopStack "IGCM_sys_ncflint"
2894	}
2895
2896	#D-#==================================================
2897	#D-function IGCM_sys_ncks
2898	#D-* Purpose: encapsulate ncks call so as to manage error code and retry
2899	#D-* Examples:
2900	#D-
2901	function IGCM_sys_ncks {
2902	IGCM_debug_PushStack "IGCM_sys_ncks" $@
2903	if ( $DEBUG_sys ) ; then
2904	echo "IGCM_sys_ncks :" $@
2905	fi
2906
2907	typeset NB_ESSAI DELAI status i
2908	# number of tentative
2909	NB_ESSAI=3
2910	# time delay between tentative
2911	DELAI=2
2912
2913	i=0
2914	while [ $i -lt $NB_ESSAI ] ; do
2915	ncks -C $@ > /tmp/out_command.$$ 2>&1
2916	status=$?
2917	if [ ${status} -gt 0 ] ; then
2918	IGCM_debug_Print 2 "IGCM_sys_ncks : error code ${status}"
2919	cat /tmp/out_command.$$
2920	\rm /tmp/out_command.$$
2921	IGCM_debug_Print 2 "IGCM_sys_ncks : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2922	sleep $DELAI
2923	else
2924	\rm /tmp/out_command.$$
2925	break
2926	fi
2927	(( i = i + 1 ))
2928	done
2929
2930	if [ ${status} -gt 0 ] ; then
2931	echo "IGCM_sys_ncks : ncks error"
2932	IGCM_debug_Exit "ncks"
2933	fi
2934
2935	IGCM_debug_PopStack "IGCM_sys_ncks"
2936	}
2937
2938	#D-#==================================================
2939	#D-function IGCM_sys_ncpdq
2940	#D-* Purpose: encapsulate ncpdq call so as to manage error code and retry
2941	#D-* Examples:
2942	#D-
2943	function IGCM_sys_ncpdq {
2944	IGCM_debug_PushStack "IGCM_sys_ncpdq" $@
2945	if ( $DEBUG_sys ) ; then
2946	echo "IGCM_sys_ncpdq :" $@
2947	fi
2948
2949	typeset NB_ESSAI DELAI status i
2950	# number of tentative
2951	NB_ESSAI=3
2952	# time delay between tentative
2953	DELAI=2
2954
2955	i=0
2956	while [ $i -lt $NB_ESSAI ] ; do
2957	ncpdq -C $@ > /tmp/out_command.$$ 2>&1
2958	status=$?
2959	if [ ${status} -gt 0 ] ; then
2960	IGCM_debug_Print 2 "IGCM_sys_ncpdq : error code ${status}"
2961	cat /tmp/out_command.$$
2962	\rm /tmp/out_command.$$
2963	IGCM_debug_Print 2 "IGCM_sys_ncpdq : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2964	sleep $DELAI
2965	else
2966	\rm /tmp/out_command.$$
2967	break
2968	fi
2969	(( i = i + 1 ))
2970	done
2971
2972	if [ ${status} -gt 0 ] ; then
2973	echo "IGCM_sys_ncpdq : ncpdq error"
2974	IGCM_debug_Exit "ncpdq"
2975	fi
2976
2977	IGCM_debug_PopStack "IGCM_sys_ncpdq"
2978	}
2979
2980	#D-#==================================================
2981	#D-function IGCM_sys_ncra
2982	#D-* Purpose: encapsulate ncra call so as to manage error code and retry
2983	#D-* Examples:
2984	#D-
2985	function IGCM_sys_ncra {
2986	IGCM_debug_PushStack "IGCM_sys_ncra" $@
2987	if ( $DEBUG_sys ) ; then
2988	echo "IGCM_sys_ncra :" $@
2989	fi
2990
2991	typeset NB_ESSAI DELAI status i
2992	# number of tentative
2993	NB_ESSAI=3
2994	# time delay between tentative
2995	DELAI=2
2996
2997	i=0
2998	while [ $i -lt $NB_ESSAI ] ; do
2999	ncra -C $@ > /tmp/out_command.$$ 2>&1
3000	status=$?
3001	if [ ${status} -gt 0 ] ; then
3002	IGCM_debug_Print 2 "IGCM_sys_ncra : error code ${status}"
3003	cat /tmp/out_command.$$
3004	\rm /tmp/out_command.$$
3005	IGCM_debug_Print 2 "IGCM_sys_ncra : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3006	sleep $DELAI
3007	else
3008	\rm /tmp/out_command.$$
3009	break
3010	fi
3011	(( i = i + 1 ))
3012	done
3013
3014	if [ ${status} -gt 0 ] ; then
3015	echo "IGCM_sys_ncra : ncra error"
3016	IGCM_debug_Exit "ncra"
3017	fi
3018
3019	IGCM_debug_PopStack "IGCM_sys_ncra"
3020	}
3021
3022	#D-#==================================================
3023	#D-function IGCM_sys_ncrcat
3024	#D-* Purpose: encapsulate ncrcat call so as to manage error code and retry
3025	#D-* Examples:
3026	#D-
3027	function IGCM_sys_ncrcat {
3028	IGCM_debug_PushStack "IGCM_sys_ncrcat" $@
3029	if ( $DEBUG_sys ) ; then
3030	echo "IGCM_sys_ncrcat :" $@
3031	fi
3032
3033	typeset NB_ESSAI DELAI status i lastArg
3034	# number of tentative
3035	NB_ESSAI=3
3036	# time delay between tentative
3037	DELAI=2
3038
3039	i=0
3040	while [ $i -lt $NB_ESSAI ] ; do
3041	ncrcat -C $@ > /tmp/out_command.$$ 2>&1
3042	status=$?
3043	if [ ${status} -gt 0 ] ; then
3044	IGCM_debug_Print 2 "IGCM_sys_ncrcat : error code ${status}"
3045	cat /tmp/out_command.$$
3046	\rm /tmp/out_command.$$
3047	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3048	sleep $DELAI
3049	elif [ ! "X$( grep "WARNING Intra-file non-monotonicity" /tmp/out_command.$$ )" = "X" ] ; then
3050	IGCM_debug_Print 2 "IGCM_sys_ncrcat : WARNING Intra-file non-monotonicity"
3051	cat /tmp/out_command.$$
3052	# remove files having corrupted time axis
3053	eval lastArg=\${$#}
3054	IGCM_debug_Print 2 "IGCM_sys_ncrcat : Delete ${lastArg}"
3055	\rm ${lastArg}
3056	\rm /tmp/out_command.$$
3057	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3058	sleep $DELAI
3059	else
3060	\rm /tmp/out_command.$$
3061	break
3062	fi
3063	(( i = i + 1 ))
3064	done
3065
3066	if [ ${status} -gt 0 ] ; then
3067	echo "IGCM_sys_ncrcat : ncrcat error"
3068	#IGCM_debug_Exit "ncrcat"
3069	fi
3070
3071	IGCM_debug_PopStack "IGCM_sys_ncrcat"
3072	}
3073
3074	#D-#==================================================
3075	#D-function IGCM_sys_ncrename
3076	#D-* Purpose: encapsulate ncrename call so as to manage error code and retry
3077	#D-* Examples:
3078	#D-
3079	function IGCM_sys_ncrename {
3080	IGCM_debug_PushStack "IGCM_sys_ncrename" $@
3081	if ( $DEBUG_sys ) ; then
3082	echo "IGCM_sys_ncrename :" $@
3083	fi
3084
3085	typeset NB_ESSAI DELAI status i
3086	# number of tentative
3087	NB_ESSAI=3
3088	# time delay between tentative
3089	DELAI=2
3090
3091	i=0
3092	while [ $i -lt $NB_ESSAI ] ; do
3093	ncrename $@ > /tmp/out_command.$$ 2>&1
3094	status=$?
3095	if [ ${status} -gt 0 ] ; then
3096	IGCM_debug_Print 2 "IGCM_sys_ncrename : error code ${status}"
3097	cat /tmp/out_command.$$
3098	\rm /tmp/out_command.$$
3099	IGCM_debug_Print 2 "IGCM_sys_ncrename : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3100	sleep $DELAI
3101	else
3102	\rm /tmp/out_command.$$
3103	break
3104	fi
3105	(( i = i + 1 ))
3106	done
3107
3108	if [ ${status} -gt 0 ] ; then
3109	echo "IGCM_sys_ncrename : ncrename error"
3110	IGCM_debug_Exit "ncrename"
3111	fi
3112
3113	IGCM_debug_PopStack "IGCM_sys_ncrename"
3114	}
3115
3116	#D-#==================================================
3117	#D-function IGCM_sys_ncwa
3118	#D-* Purpose: encapsulate ncwa call so as to manage error code and retry
3119	#D-* Examples:
3120	#D-
3121	function IGCM_sys_ncwa {
3122	IGCM_debug_PushStack "IGCM_sys_ncwa" $@
3123	if ( $DEBUG_sys ) ; then
3124	echo "IGCM_sys_ncwa :" $@
3125	fi
3126
3127	typeset NB_ESSAI DELAI status i
3128	# number of tentative
3129	NB_ESSAI=3
3130	# time delay between tentative
3131	DELAI=2
3132
3133	i=0
3134	while [ $i -lt $NB_ESSAI ] ; do
3135	ncwa -C $@ > /tmp/out_command.$$ 2>&1
3136	status=$?
3137	if [ ${status} -gt 0 ] ; then
3138	IGCM_debug_Print 2 "IGCM_sys_ncwa : error code ${status}"
3139	cat /tmp/out_command.$$
3140	\rm /tmp/out_command.$$
3141	IGCM_debug_Print 2 "IGCM_sys_ncwa : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3142	sleep $DELAI
3143	else
3144	\rm /tmp/out_command.$$
3145	break
3146	fi
3147	(( i = i + 1 ))
3148	done
3149
3150	if [ ${status} -gt 0 ] ; then
3151	echo "IGCM_sys_ncwa : ncwa error"
3152	IGCM_debug_Exit "ncwa"
3153	fi
3154
3155	IGCM_debug_PopStack "IGCM_sys_ncwa"
3156	}
3157
3158	##############################################################
3159	# CDO OPERATOR
3160
3161	#D-#==================================================
3162	#D-function IGCM_sys_cdo
3163	#D-* Purpose: encapsulate cdo call so as to manage error code and retry
3164	#D-* Examples:
3165	#D-
3166	function IGCM_sys_cdo {
3167	IGCM_debug_PushStack "IGCM_sys_cdo" $@
3168	if ( $DEBUG_sys ) ; then
3169	echo "IGCM_sys_cdo :" $@
3170	fi
3171
3172	typeset status
3173
3174	\cdo $@ > /tmp/out_command.$$ 2>&1
3175	status=$?
3176	if [ ${status} -gt 0 ] ; then
3177	echo "IGCM_sys_cdo : error code ${status}"
3178	cat /tmp/out_command.$$
3179	\rm /tmp/out_command.$$
3180	IGCM_debug_PopStack "IGCM_sys_cdo"
3181	return 1
3182	else
3183	IGCM_debug_PopStack "IGCM_sys_cdo"
3184	return 0
3185	fi
3186
3187	IGCM_debug_PopStack "IGCM_sys_cdo"
3188	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: