Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_curie.ksh @ 1150

Last change on this file since 1150 was 1150, checked in by sdipsl, 9 years ago
adressing #242 message 2000 and 3000 (pushstack and popstack) has been turned off for the moment to ease downstream processing
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 93.3 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Curie
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=curie
71	# add default project on curie
72	typeset DEFAULT_PROJECT=gen0826
73
74	#D-
75	#D-#==================================================
76	#D-Program used in libIGCM
77	#D-#==================================================
78
79	# Submit command
80	typeset SUBMIT=${SUBMIT:=ccc_msub}
81	# rsync with path
82	typeset -r RSYNC=/usr/bin/rsync
83	# RSYNC_opt args to rsync
84	typeset -r RSYNC_opt="-va"
85	# ie storage filesystem
86	typeset -r STOREHOST=${MASTER}
87
88	#====================================================
89	# Source default environment
90	#====================================================
91	##. /etc/profile
92
93	#====================================================
94	# Set environment tools (ferret, nco, cdo)
95	#====================================================
96	if [ X${TaskType} = Xcomputing ] ; then
97	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
98	# to run with netcdf 3.6.3 ie compilation done before 17/2/2014
99	# uncomment 2 lines :
100	# module unload netcdf
101	# module load netcdf/3.6.3
102	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddNoise/src_X64_CURIE/bin
103	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddPerturbation/src_X64_CURIE/bin
104	else
105	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
106	fi
107
108	#====================================================
109	# Specific for ocean additionnal diagnostic
110	export FER_GO="$FER_GO /home/cont003/p86denv/IGCM_POST_UTIL/JNL /home/cont003/p86denv/GRAF /home/cont003/p86denv/GRAF/GO"
111	export FER_PALETTE="$FER_PALETTE /home/cont003/p86denv/GRAF/PALET"
112
113	#====================================================
114	# Host specific DIRECTORIES
115	#====================================================
116
117	# ============ CESIUM START ============ #
118
119	#====================================================
120	#- Mirror libIGCM from titane to cesium if needed
121	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
122	#if [ ! ${ROOTSYS} = "home" ] ; then
123	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
124	#else
125	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
126	#fi
127
128	#====================================================
129	#- libIGCM_POST
130	#if ( ${MirrorlibIGCM} ) ; then
131	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
132	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
133	#else
134	# typeset -r libIGCM_POST=${libIGCM}
135	#fi
136
137	# ============ CESIUM END ============ #
138
139	#====================================================
140	#- MirrorlibIGCM for frontend
141	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
142
143	#====================================================
144	#- libIGCM_POST for frontend
145	typeset -r libIGCM_POST=${libIGCM}
146
147	#====================================================
148	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
149	typeset -r R_EXE="${MODIPSL}/bin"
150
151	#====================================================
152	#- SUBMIT_DIR : submission dir
153	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
154
155	#====================================================
156	#- IN
157	typeset -r R_IN=${R_IN:=/ccc/work/cont003/dsm/p86ipsl/IGCM}
158	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/ccc/work/cont003/dsm/p24data}
159
160	#====================================================
161	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
162	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
163
164	#====================================================
165	#- HOST_MPIRUN_COMMAND
166	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1"}
167
168	#====================================================
169	#- Max number of arguments passed to nco operator or demigration command
170	UNIX_MAX_LIMIT=360
171
172	#====================================================
173	#- set PackDefault true on curie
174	PackDefault=true
175
176	#====================================================
177	#- Number of core per node (max number of OpenMP task)
178	NUM_COREPERNODE=8
179
180	#====================================================
181	#- Default number of MPI task for IPSL coupled model
182	#- required for backward compatibility
183	#-
184	DEFAULT_NUM_PROC_OCE=5
185	DEFAULT_NUM_PROC_CPL=1
186	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
187	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
188
189	#D-#==================================================
190	#D-function IGCM_sys_defineArchives
191	#D-* Purpose:
192	#D-* Define ARCHIVE : Dedicated to large files
193	#D-* Define STORAGE : Dedicated to small/medium files
194	#D-* Define R_OUT : Output tree located on ARCHIVE
195	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
196	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
197	#D-* if SpaceName=TEST everything is stored on SCRATCHDIR
198	#D-* Examples:
199	#D-
200	function IGCM_sys_defineArchives {
201	IGCM_debug_PushStack "IGCM_sys_defineArchives"
202
203	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
204	#====================================================
205	#- ARCHIVE (dedicated to large files)
206	ARCHIVE=${config_UserChoices_ARCHIVE}
207	else
208	#====================================================
209	#- ARCHIVE (dedicated to large files)
210	ARCHIVE=${CCCSTOREDIR}
211	fi
212
213	if [ ! X${config_UserChoices_STORAGE} = X ]; then
214	#====================================================
215	#- STORAGE (dedicated to small/medium files)
216	STORAGE=${config_UserChoices_STORAGE}
217	else
218	#====================================================
219	#- STORAGE (dedicated to small/medium files)
220	STORAGE=${CCCWORKDIR}
221	fi
222
223	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
224	#====================================================
225	#- R_OUT
226	R_OUT=${SCRATCHDIR}/IGCM_OUT
227
228	#====================================================
229	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
230	R_FIG=${SCRATCHDIR}/IGCM_OUT
231
232	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
233
234	else
235	#====================================================
236	#- R_OUT
237	R_OUT=${ARCHIVE}/IGCM_OUT
238
239	#====================================================
240	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
241	R_FIG=${STORAGE}/IGCM_OUT
242	fi
243
244	#====================================================
245	#- R_BUF (ONLY FOR double copy an scratch)
246	R_BUF=${SCRATCHDIR}/IGCM_OUT
247
248	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
249	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
250	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
251
252	IGCM_debug_PopStack "IGCM_sys_defineArchives"
253	}
254
255	#D-#==================================================
256	#D-function IGCM_sys_RshMaster
257	#D-* Purpose: Connection to frontend machine.
258	#D-* Examples:
259	#D-
260	function IGCM_sys_RshMaster {
261	IGCM_debug_PushStack "IGCM_sys_RshMaster" $@
262	/bin/ksh <<-EOF
263	export libIGCM=${libIGCM}
264	export DEBUG_debug=${DEBUG_debug}
265	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
266	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
267	${@}
268	EOF
269	if [ $? -gt 0 ] ; then
270	echo "IGCM_sys_RshMaster : erreur."
271	IGCM_debug_Exit "IGCM_sys_RshMaster"
272	fi
273	IGCM_debug_PopStack "IGCM_sys_RshMaster"
274	}
275
276	#D-#==================================================
277	#D-function IGCM_sys_RshArchive
278	#D-* Purpose: Archive rsh command
279	#D-* Examples:
280	#D-
281	function IGCM_sys_RshArchive {
282	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
283	/bin/ksh <<-EOF
284	${@}
285	EOF
286	status=$?
287	if [ ${status} -gt 0 ] ; then
288	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
289	IGCM_debug_Exit "IGCM_sys_RshArchive"
290	fi
291	IGCM_debug_PopStack "IGCM_sys_RshArchive"
292	}
293
294	#D-#==================================================
295	#D-function IGCM_sys_RshArchive_NoError
296	#D-* Purpose: Archive rsh command, without error
297	#D-* used only in monitoring.job
298	#D-* Examples:
299	#D-
300	function IGCM_sys_RshArchive_NoError {
301	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
302	/bin/ksh <<-EOF
303	${@} 2> \dev\null
304	EOF
305	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
306	}
307
308	#D-#==================================================
309	#D-function IGCM_sys_RshPost
310	#D-* Purpose: Post-process rsh command
311	#D-* Examples:
312	#D-
313	function IGCM_sys_RshPost {
314	IGCM_debug_PushStack "IGCM_sys_RshPost" $@
315	if ( $DEBUG_sys ) ; then
316	echo "IGCM_sys_RshPost :" $@
317	fi
318	# keep standard input (stdin) for the loop onto temporary file
319	cat >/tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
320
321	/bin/ksh </tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
322	if [ $? -gt 0 ] ; then
323	echo "IGCM_sys_RshPost : erreur."
324	IGCM_debug_Exit "IGCM_sys_RshPost"
325	fi
326	# delete temporary file
327	\rm /tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
328
329	# ============ FRONTEND END ============ #
330
331	# ============ CESIUM START ============ #
332	# typeset NB_ESSAI DELAI status i
333	# if [ "X$( grep rebuild_from tmp_IGCM_sys_RshPost_$$ )" != "X" ] ; then
334	# #little hack so that rebuild submission is done on titane not an cesium
335	#
336	# libIGCM_POST_sed=$( echo $libIGCM_POST \| sed 's/\//\\\//g' )
337	# POST_DIR_sed=$( echo ${POST_DIR} \| sed 's/\//\\\//g' )
338	# sed "s/IGCM_sys_QsubPost/IGCM_sys_Qsub/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
339	# sed "s/ rebuild_fromWorkdir/ ${libIGCM_POST_sed}\/rebuild_fromWorkdir.job/g" tmp.txt > tmp_IGCM_sys_RshPost_$$
340	# sed "s/ rebuild_fromArchive/ ${libIGCM_POST_sed}\/rebuild_fromArchive.job/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
341	# sed "s/Script_Post_Output=/Script_Output=${POST_DIR_sed}\//g" tmp.txt > tmp_IGCM_sys_RshPost_$$
342	# \mv tmp.txt tmp_IGCM_sys_RshPost_$$
343	#
344	# echo cat tmp_IGCM_sys_RshPost_$$ AFTER
345	# cat tmp_IGCM_sys_RshPost_$$
346	#
347	# /bin/ksh <tmp_IGCM_sys_RshPost_$$
348	# if [ $? -gt 0 ] ; then
349	# echo "IGCM_sys_RshPost : erreur."
350	# IGCM_debug_Exit "IGCM_sys_RshPost"
351	# fi
352	# # delete temporary file
353	# \rm tmp_IGCM_sys_RshPost_$$
354	#
355	# else
356	# # number of tentative
357	# NB_ESSAI=10
358	# # time delay between tentative
359	# DELAI=10
360	# i=0
361	# while [ $i -ne $NB_ESSAI ] ; do
362	# ssh -t titane996 ssh cesium /bin/ksh <tmp_IGCM_sys_RshPost_$$
363	# status=$?
364	# if [ ${status} -ne 0 ];
365	# then
366	# sleep $DELAI
367	# else
368	# break
369	# fi
370	# let i=$i+1
371	# done
372	# # delete temporary file
373	# \rm tmp_IGCM_sys_RshPost_$$
374	#
375	# if [ ${status} -gt 0 ] ; then
376	# echo "IGCM_sys_RshPost : erreur."
377	# IGCM_debug_Exit "IGCM_sys_RshPost"
378	# fi
379	# fi
380
381	# ============ CESIUM END ============ #
382
383	IGCM_debug_PopStack "IGCM_sys_RshPost"
384	}
385
386	#D-#==================================================
387	#D-function IGCM_sys_SendMail
388	#D-* Purpose: Send mail when simulation is over
389	#D-* Examples:
390	#D-
391	function IGCM_sys_SendMail {
392	IGCM_debug_PushStack "IGCM_sys_SendMail" $@
393
394	if ( $DEBUG_sys ) ; then
395	echo "IGCM_sys_SendMail :" $@
396	fi
397
398	if [ X${1} = XAccounting ] ; then
399	status=Accounting
400	mailText=jobAccounting.mail
401	elif ( ${ExitFlag} ) ; then
402	status=failed
403	mailText=jobEnd.mail
404	else
405	status=completed
406	mailText=jobEnd.mail
407	fi
408
409	# Update selected mail template
410	while read -r line; do
411	eval echo $line >> mail.txt ;
412	done < ${libIGCM}/libIGCM_sys/${mailText}
413
414	if [ ! -z ${config_UserChoices_MailName} ] ; then
415	mail -s "${config_UserChoices_JobName} ${status}" ${config_UserChoices_MailName} < mail.txt
416	elif [ -f ~/.forward ] ; then
417	mail -s "${config_UserChoices_JobName} ${status}" $( cat ~/.forward ) < mail.txt
418	fi
419
420	sleep 10
421	rm -f mail.txt
422
423	IGCM_debug_PopStack "IGCM_sys_SendMail"
424	}
425
426	#D-#==================================================
427	#D-function IGCM_sys_Mkdir
428	#D-* Purpose: Master locale mkdir command
429	#D-* Examples:
430	#D-
431	function IGCM_sys_Mkdir {
432	IGCM_debug_PushStack "IGCM_sys_Mkdir" $@
433	if ( $DEBUG_sys ) ; then
434	echo "IGCM_sys_Mkdir :" $@
435	fi
436	if [ ! -d ${1} ]; then
437	\mkdir -p $1
438	if [ $? -gt 0 ] ; then
439	echo "IGCM_sys_Mkdir : erreur."
440	IGCM_debug_Exit "IGCM_sys_Mkdir"
441	fi
442	fi
443	# vérification :
444	if [ ! -d ${1} ] ; then
445	echo "IGCM_sys_Mkdir : erreur."
446	IGCM_debug_Exit "IGCM_sys_Mkdir"
447	fi
448	IGCM_debug_PopStack "IGCM_sys_Mkdir"
449	}
450
451	#D-#==================================================
452	#D-function IGCM_sys_MkdirArchive
453	#D-* Purpose: Mkdir on Archive
454	#D-* Examples:
455	#D-
456	function IGCM_sys_MkdirArchive {
457	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
458	if ( $DEBUG_sys ) ; then
459	echo "IGCM_sys_MkdirArchive :" $@
460	fi
461	#- creation de repertoire sur le serveur fichier
462	if [ ! -d ${1} ]; then
463	\mkdir -p $1
464	status=$?
465
466	if [ ${status} -gt 0 ] ; then
467	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
468	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
469	fi
470	fi
471	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
472	}
473
474	#D-#==================================================
475	#D-function IGCM_sys_MkdirWork
476	#D-* Purpose: Mkdir on Work
477	#D-* Examples:
478	#D-
479	function IGCM_sys_MkdirWork {
480	IGCM_debug_PushStack "IGCM_sys_MkdirWork" $@
481	if ( $DEBUG_sys ) ; then
482	echo "IGCM_sys_MkdirWork :" $@
483	fi
484	#- creation de repertoire sur le serveur fichier
485	if [ ! -d ${1} ]; then
486	\mkdir -p $1
487	if [ $? -gt 0 ] ; then
488	echo "IGCM_sys_MkdirWork : erreur."
489	IGCM_debug_Exit "IGCM_sys_MkdirWork"
490	fi
491	fi
492	IGCM_debug_PopStack "IGCM_sys_MkdirWork"
493	}
494
495	#D-#==================================================
496	#D-function IGCM_sys_Cd
497	#D-* Purpose: master cd command
498	#D-* Examples:
499	#D-
500	function IGCM_sys_Cd {
501	IGCM_debug_PushStack "IGCM_sys_Cd" $@
502	if ( $DEBUG_sys ) ; then
503	echo "IGCM_sys_Cd :" $@
504	fi
505	\cd $1
506	if [ $? -gt 0 ] ; then
507	echo "IGCM_sys_Cd : erreur."
508	IGCM_debug_Exit "IGCM_sys_Cd"
509	fi
510	IGCM_debug_PopStack "IGCM_sys_Cd"
511	}
512
513	#D-#==================================================
514	#D-function IGCM_sys_Chmod
515	#D-* Purpose: Chmod
516	#D-* Examples:
517	#D-
518	function IGCM_sys_Chmod {
519	IGCM_debug_PushStack "IGCM_sys_Chmod" $@
520	if ( $DEBUG_sys ) ; then
521	echo "IGCM_sys_Chmod :" $@
522	fi
523	\chmod $@
524	if [ $? -gt 0 ] ; then
525	echo "IGCM_sys_Chmod : erreur."
526	IGCM_debug_Exit "IGCM_sys_Chmod"
527	fi
528	IGCM_debug_PopStack "IGCM_sys_Chmod"
529	}
530
531	#D-#==================================================
532	#D-function IGCM_sys_FileSize
533	#D-* Purpose: Filesize
534	#D-* Examples:
535	#D-
536	function IGCM_sys_FileSize {
537	IGCM_debug_PushStack "IGCM_sys_FileSize" $@
538
539	typeset sizeF
540	set +A sizeF -- $( ls -la ${1} )
541	if [ $? -gt 0 ] ; then
542	IGCM_debug_Exit "IGCM_sys_FileSize"
543	fi
544	eval ${2}=${sizeF[4]}
545
546	IGCM_debug_PopStack "IGCM_sys_FileSize"
547	}
548
549	#D-#==================================================
550	#D-function IGCM_sys_TestDir
551	#D-* Purpose: Test Directory that must exists
552	#D-* Examples:
553	#D-
554	function IGCM_sys_TestDir {
555	IGCM_debug_PushStack "IGCM_sys_TestDir" $@
556	if ( $DEBUG_sys ) ; then
557	echo "IGCM_sys_TestDir :" $@
558	fi
559	typeset ExistFlag
560	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
561	IGCM_debug_PopStack "IGCM_sys_TestDir"
562
563	return ${ExistFlag}
564	}
565
566	#D-#==================================================
567	#D-function IGCM_sys_TestDirArchive
568	#D-* Purpose: Test Directory that must exists on Archive
569	#D-* Examples:
570	#D-
571	function IGCM_sys_TestDirArchive {
572	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
573	if ( $DEBUG_sys ) ; then
574	echo "IGCM_sys_TestDirArchive :" $@
575	fi
576	typeset ExistFlag
577	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
578	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
579	return ${ExistFlag}
580	}
581
582	#D-#==================================================
583	#D-function IGCM_sys_IsFileArchived
584	#D-* Purpose: Test file that must NOT EXISTS on Archive
585	#D-* Examples:
586	#D-
587	function IGCM_sys_IsFileArchived {
588	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
589	if ( $DEBUG_sys ) ; then
590	echo "IGCM_sys_IsFileArchived :" $@
591	fi
592	typeset IsArchivedFlag
593	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
594	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
595
596	return ${IsArchivedFlag}
597	}
598
599	#D-#==================================================
600	#D-function IGCM_sys_TestFileArchive
601	#D-* Purpose: Test file that must NOT EXISTS on Archive
602	#D-* Examples:
603	#D-
604	function IGCM_sys_TestFileArchive {
605	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
606	typeset ExistFlag
607	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
608	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
609
610	return ${ExistFlag}
611	}
612
613	#D-#==================================================
614	#D-function IGCM_sys_TestFileBuffer
615	#D-* Purpose: Test file that must NOT EXISTS on Buffer
616	#D-* Examples:
617	#D-
618	function IGCM_sys_TestFileBuffer {
619	IGCM_debug_PushStack "IGCM_sys_TestFileBuffer" $@
620	typeset ExistFlag
621	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
622	IGCM_debug_PopStack "IGCM_sys_TestFileBuffer"
623
624	return ${ExistFlag}
625	}
626
627	#D-#==================================================
628	#D-function IGCM_sys_CountFileArchive
629	#D-* Purpose: Count files on Archive filesystem
630	#D-* Examples:
631	#D-
632	function IGCM_sys_CountFileArchive {
633	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
634	ls ${@} 2>/dev/null \| wc -l
635	if [ $? -gt 0 ] ; then
636	echo "IGCM_sys_CountFileArchive : erreur."
637	fi
638	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
639	}
640
641	#D-#==================================================
642	#D-function IGCM_sys_CountFileBuffer
643	#D-* Purpose: Count files on Scratch filesystem
644	#D-* Examples:
645	#D-
646	function IGCM_sys_CountFileBuffer {
647	IGCM_debug_PushStack "IGCM_sys_CountFileBuffer" $@
648	ls ${@} 2>/dev/null \| wc -l
649	if [ $? -gt 0 ] ; then
650	echo "IGCM_sys_CountFileBuffer : erreur."
651	fi
652	IGCM_debug_PopStack "IGCM_sys_CountFileBuffer"
653	}
654
655	#D-#==================================================
656	#D-function IGCM_sys_Tree
657	#D-* Purpose: Tree directories with files on ${ARCHIVE}
658	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
659	#D-
660	function IGCM_sys_Tree {
661	IGCM_debug_PushStack "IGCM_sys_Tree" $@
662	if ( $DEBUG_sys ) ; then
663	echo "IGCM_sys_Tree :" $@
664	fi
665
666	\ls -lR ${@}
667
668	IGCM_debug_PopStack "IGCM_sys_Tree"
669	}
670
671	#D-#==================================================
672	#D-function IGCM_sys_Tar
673	#D-* Purpose: master tar command
674	#D-* Examples:
675	#D-
676	function IGCM_sys_Tar {
677	IGCM_debug_PushStack "IGCM_sys_Tar" $@
678	if ( $DEBUG_sys ) ; then
679	echo "IGCM_sys_Tar :" $@
680	fi
681	\tar cf $@
682	if [ $? -gt 0 ] ; then
683	echo "IGCM_sys_Tar : erreur."
684	IGCM_debug_Exit "IGCM_sys_Tar"
685	fi
686	IGCM_debug_PopStack "IGCM_sys_Tar"
687	}
688
689	#D-#==================================================
690	#D-function IGCM_sys_UnTar
691	#D-* Purpose: master un-tar command
692	#D-* Examples:
693	#D-
694	function IGCM_sys_UnTar {
695	IGCM_debug_PushStack "IGCM_sys_UnTar" $@
696	if ( $DEBUG_sys ) ; then
697	echo "IGCM_sys_UnTar :" $@
698	fi
699	\tar xvf $1
700	if [ $? -gt 0 ] ; then
701	echo "IGCM_sys_UnTar : erreur."
702	IGCM_debug_Exit "IGCM_sys_UnTar"
703	fi
704	IGCM_debug_PopStack "IGCM_sys_UnTar"
705	}
706
707	#D-#==================================================
708	#D-function IGCM_sys_Qsub
709	#D-* Purpose: Qsub new job
710	#D-* Examples:
711	#D-
712	function IGCM_sys_Qsub {
713	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
714	if ( $DEBUG_sys ) ; then
715	echo "IGCM_sys_Qsub :" $@
716	fi
717	typeset options status
718	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
719
720	/usr/bin/ccc_msub ${options} $1 > /tmp/out_command.$$ 2>&1
721	status=$?
722
723	cat /tmp/out_command.$$
724	if [ ${status} -gt 0 ] ; then
725	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
726	IGCM_debug_Exit "IGCM_sys_Qsub"
727	else
728	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
729	fi
730	IGCM_debug_PopStack "IGCM_sys_Qsub"
731	}
732
733	#D-#==================================================
734	#D-function IGCM_sys_QsubPost
735	#D-* Purpose: Qsub new job on scalaire
736	#D-* Examples:
737	#D-
738	function IGCM_sys_QsubPost {
739	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
740	if ( $DEBUG_sys ) ; then
741	echo "IGCM_sys_QsubPost :" $@
742	fi
743	typeset options status
744	MY_PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=${DEFAULT_PROJECT}} \| cut -d@ -f1 )
745
746	# EASIER TO DO THIS INSTEAD OF DUPLICATING libIGCM_sys_curie.ksh
747	case $( hostname -s ) in
748	curie*)
749	options="-Q normal -A ${MY_PROJECT} -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out";;
750	airain*)
751	options="-q ivybridge -A dsm -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
752	esac
753
754	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > /tmp/out_command.$$ 2>&1
755	status=$?
756
757	cat /tmp/out_command.$$
758	if [ ${status} -gt 0 ] ; then
759	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
760	IGCM_debug_Exit "IGCM_sys_QsubPost"
761	else
762	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
763	fi
764	IGCM_debug_PopStack "IGCM_sys_QsubPost"
765	}
766
767	#D-*************************
768	#D- File transfer functions
769	#D-*************************
770	#D-
771
772	#D-#==================================================
773	#D-function IGCM_sys_Rsync_out
774	#D-* Purpose: treat return val of rsync
775	#D-* Examples: IGCM_sys_Rsync_out out_RET_rsync
776	#D- Error values and explanations can depend on your system version.
777	function IGCM_sys_Rsync_out {
778	status=$1
779	if [ ! $status ] ; then
780	echo "rsync error !"
781	fi
782
783	if [ $MYLANG = "fr" ]; then
784	case $status in
785	0) return ;;
786	1) echo "Erreur de rsync ; RERR_SYNTAX : "
787	echo "Erreur de syntaxe ou d'utilisation."
788	return;;
789	2) echo "Erreur de rsync ; RERR_PROTOCOL : "
790	echo "Incompatibilité de protocole."
791	return;;
792	3) echo "Erreur de rsync ; RERR_FILESELECT 3"
793	echo "Erreurs lors de la sélection des fichiers d'entrée sortie et"
794	echo "répertoires"
795	return;;
796	4) echo "Erreur de rsync ; RERR_UNSUPPORTED"
797	echo "Action demandée non supportée : une tentative de manipulation de"
798	echo "fichiers 64-bits sur une plate-forme qui ne les supporte pas a"
799	echo "été faite ; ou une option qui est supportée par le client mais"
800	echo "pas par le serveur a été spécifiée."
801	return;;
802	10) echo "Erreur de rsync ; RERR_SOCKETIO"
803	echo "Erreur dans le socket d'entrée sortie"
804	return;;
805	11) echo "Erreur de rsync ; RERR_FILEIO"
806	echo "Erreur d'entrée sortie fichier"
807	return;;
808	12) echo "Erreur de rsync ; RERR_STREAMIO"
809	echo "Erreur dans flux de donnée du protocole rsync"
810	return;;
811	13) echo "Erreur de rsync ; RERR_MESSAGEIO"
812	echo "Erreur avec les diagnostics du programme"
813	return;;
814	14) echo "Erreur de rsync ; RERR_IPC"
815	echo "Erreur dans le code IPC"
816	return;;
817	20) echo "Erreur de rsync ; RERR_SIGNAL"
818	echo "SIGUSR1 ou SIGINT reçu"
819	return;;
820	21) echo "Erreur de rsync ; RERR_WAITCHILD"
821	echo "Une erreur retournée par waitpid()"
822	return;;
823	22) echo "Erreur de rsync ; RERR_MALLOC"
824	echo "Erreur lors de l'allocation des tampons de mémoire de coeur"
825	return;;
826	23) echo ""
827	echo "Erreur fichier inexistant"
828	return;;
829	30) echo "Erreur de rsync ; RERR_TIMEOUT"
830	echo "Temps d'attente écoulé dans l'envoi/réception de données"
831	return;;
832	*) echo "Erreur de rsync : code de retour de rsync inconnu :" $status
833	return;;
834	esac
835	elif [ $MYLANG = "en" ] ; then
836	case $status in
837	0) return;;
838	1) echo "rsync error : Syntax or usage error "
839	return;;
840	2) echo "rsync error : Protocol incompatibility "
841	return;;
842	3) echo "rsync error : Errors selecting input/output files, dirs"
843	return;;
844	4) echo "rsync error : Requested action not supported: an attempt"
845	echo "was made to manipulate 64-bit files on a platform that cannot support"
846	echo "them; or an option was specified that is supported by the client and"
847	echo "not by the server."
848	return;;
849	5) echo "rsync error : Error starting client-server protocol"
850	return;;
851	10) echo "rsync error : Error in socket I/O "
852	return;;
853	11) echo "rsync error : Error in file I/O "
854	return;;
855	12) echo "rsync error : Error in rsync protocol data stream "
856	return;;
857	13) echo "rsync error : Errors with program diagnostics "
858	return;;
859	14) echo "rsync error : Error in IPC code "
860	return;;
861	20) echo "rsync error : Received SIGUSR1 or SIGINT "
862	return;;
863	21) echo "rsync error : Some error returned by waitpid() "
864	return;;
865	22) echo "rsync error : Error allocating core memory buffers "
866	return;;
867	23) echo "rsync error : Partial transfer due to error"
868	return;;
869	24) echo "rsync error : Partial transfer due to vanished source files"
870	return;;
871	30) echo "rsync error : Timeout in data send/receive "
872	return;;
873	*) echo "rsync error : return code of rsync unknown :" $status
874	return;;
875	esac
876	else
877	echo "unknown language $MYLANG."
878	return
879	fi
880	}
881
882	#D-#==================================================
883	#D-function IGCM_sys_Miror_libIGCM
884	#D-* Purpose: Mirror libIGCM PATH and lib to frontend
885	#D-* Examples:
886	#D-
887	function IGCM_sys_Mirror_libIGCM {
888	IGCM_debug_PushStack "IGCM_sys_Mirror_libIGCM"
889	if ( $DEBUG_sys ) ; then
890	echo "IGCM_sys_Mirror_libIGCM"
891	fi
892
893	typeset status
894
895	mkdir -p ${HOME}/MIRROR/${PATHlibIGCM}
896
897	echo ${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} > /tmp/out_command.$$ 2>&1
898	${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} >> /tmp/out_command.$$ 2>&1
899	status=$?
900
901	if [ ${status} -gt 0 ] ; then
902	echo "IGCM_sys_Mirror_libIGCM Warning : no libIGCM on frontend."
903	cat /tmp/out_command.$$
904	fi
905	IGCM_debug_PopStack "IGCM_sys_Mirror_libIGCM"
906	}
907
908	#D-#==================================================
909	#D-function IGCM_sys_Cp
910	#D-* Purpose: generic cp
911	#D-* Examples:
912	#D-
913	function IGCM_sys_Cp {
914	IGCM_debug_PushStack "IGCM_sys_Cp" $@
915	if ( $DEBUG_sys ) ; then
916	echo "IGCM_sys_Cp :" $@
917	fi
918
919	typeset status
920
921	echo cp $@ > /tmp/out_command.$$ 2>&1
922	\cp $@ >> /tmp/out_command.$$ 2>&1
923	status=$?
924
925	if [ ${status} -gt 0 ] ; then
926	echo "IGCM_sys_Cp : error code ${status}"
927	cat /tmp/out_command.$$
928	IGCM_debug_Exit "IGCM_sys_Cp"
929	else
930	\rm /tmp/out_command.$$
931	fi
932	IGCM_debug_PopStack "IGCM_sys_Cp"
933	}
934
935	#D-#==================================================
936	#D-function IGCM_sys_Rm
937	#D-* Purpose: generic rm
938	#D-* Examples:
939	#D-
940	function IGCM_sys_Rm {
941	IGCM_debug_PushStack "IGCM_sys_Rm" $@
942	if ( $DEBUG_sys ) ; then
943	echo "IGCM_sys_Rm :" $@
944	fi
945
946	typeset status
947
948	echo rm $@ > /tmp/out_command.$$ 2>&1
949	\rm $@ >> /tmp/out_command.$$ 2>&1
950	status=$?
951
952	if [ ${status} -gt 0 ] ; then
953	echo "IGCM_sys_Rm : error code ${status}"
954	cat /tmp/out_command.$$
955	IGCM_debug_Exit "IGCM_sys_Rm"
956	else
957	\rm /tmp/out_command.$$
958	fi
959	IGCM_debug_PopStack "IGCM_sys_Rm"
960	}
961
962	#D-#==================================================
963	#D-function IGCM_sys_RmRunDir
964	#D-* Purpose: rm tmpdir (dummy function most of the time batch
965	#D- scheduler will do the job)
966	#D-* Examples:
967	#D-
968	function IGCM_sys_RmRunDir {
969	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
970	if ( $DEBUG_sys ) ; then
971	echo "IGCM_sys_RmRunDir :" $@
972	fi
973
974	typeset status
975
976	echo rm $@ > /tmp/out_command.$$ 2>&1
977	\rm $@ >> /tmp/out_command.$$ 2>&1
978	status=$?
979
980	if [ ${status} -gt 0 ] ; then
981	echo "IGCM_sys_RmRunDir : rm error code is ${status}."
982	cat /tmp/out_command.$$
983	IGCM_debug_Exit "IGCM_sys_RmRunDir"
984	else
985	\rm /tmp/out_command.$$
986	fi
987	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
988	}
989
990	#D-#==================================================
991	#D-function IGCM_sys_Mv
992	#D-* Purpose: generic move
993	#D-* Examples:
994	#D-
995	function IGCM_sys_Mv {
996	IGCM_debug_PushStack "IGCM_sys_Mv" $@
997	if ( $DEBUG_sys ) ; then
998	echo "IGCM_sys_Mv :" $@
999	fi
1000
1001	if [ $DRYRUN = 0 ]; then
1002
1003	typeset status
1004
1005	echo mv $@ > /tmp/out_command.$$ 2>&1
1006	\mv $@ >> /tmp/out_command.$$ 2>&1
1007	status=$?
1008
1009	if [ ${status} -gt 0 ] ; then
1010	echo "IGCM_sys_Mv : error code ${status}"
1011	cat /tmp/out_command.$$
1012	IGCM_debug_Exit "IGCM_sys_Mv"
1013	else
1014	\rm /tmp/out_command.$$
1015	fi
1016	fi
1017
1018	IGCM_debug_PopStack "IGCM_sys_Mv"
1019	}
1020
1021	#D-#==================================================
1022	#D-function IGCM_sys_Put_Dir
1023	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
1024	#D-* Examples:
1025	#D-
1026	function IGCM_sys_Put_Dir {
1027	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
1028	if ( $DEBUG_sys ) ; then
1029	echo "IGCM_sys_Put_Dir :" $@
1030	fi
1031	if [ $DRYRUN = 0 ]; then
1032	if [ ! -d ${1} ] ; then
1033	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
1034	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
1035	return
1036	fi
1037
1038	typeset status
1039
1040	# Only if we use rsync
1041	#IGCM_sys_TestDirArchive $( dirname $2 )
1042	#
1043	#USUAL WAY
1044	\cp -r $1 $2 > /tmp/out_command.$$ 2>&1
1045	status=$?
1046
1047	if [ ${status} -gt 0 ] ; then
1048	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
1049	cat /tmp/out_command.$$
1050	IGCM_debug_Exit "IGCM_sys_Put_Dir"
1051	else
1052	\rm /tmp/out_command.$$
1053	fi
1054	fi
1055	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
1056	}
1057
1058	#D-#==================================================
1059	#D-function IGCM_sys_Get_Dir
1060	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
1061	#D-* Examples:
1062	#D-
1063	function IGCM_sys_Get_Dir {
1064	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
1065	if ( $DEBUG_sys ) ; then
1066	echo "IGCM_sys_Get_Dir :" $@
1067	fi
1068	if [ $DRYRUN = 0 ]; then
1069	# if [ ! -d ${1} ] ; then
1070	# echo "WARNING : IGCM_sys_Get_Dir ${1} DOES NOT EXIST ."
1071	# IGCM_debug_PopStack "IGCM_sys_Get_Dir"
1072	# return
1073	# fi
1074
1075	typeset NB_ESSAI DELAI status i
1076	# number of tentative
1077	NB_ESSAI=3
1078	# time delay between tentative
1079	DELAI=2
1080
1081	# Only if we use rsync
1082	#IGCM_sys_TestDirArchive $( dirname $2 )
1083	#
1084	# USUAL WAY
1085	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
1086	ccc_hsm get -r $1
1087
1088	i=0
1089	while [ $i -lt $NB_ESSAI ] ; do
1090	\cp -ur $1 $2 >> /tmp/out_command.$$ 2>&1
1091	status=$?
1092	if [ ${status} -gt 0 ] ; then
1093	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
1094	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
1095	sleep $DELAI
1096	else
1097	break
1098	fi
1099	(( i = i + 1 ))
1100	done
1101
1102	if [ ${status} -gt 0 ] ; then
1103	echo "IGCM_sys_Get_Dir : error."
1104	cat /tmp/out_command.$$
1105	IGCM_debug_Exit "IGCM_sys_Get_Dir"
1106	else
1107	\rm /tmp/out_command.$$
1108	fi
1109	fi
1110	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
1111	}
1112
1113	#D-#==================================================
1114	#D-function IGCM_sys_Get_Master
1115	#D-* Purpose: Copy a complete directory from MASTER filesystem
1116	#D-* Examples:
1117	#D-
1118	function IGCM_sys_Get_Master {
1119	IGCM_debug_PushStack "IGCM_sys_Get_Master" $@
1120	if ( $DEBUG_sys ) ; then
1121	echo "IGCM_sys_Get_Master :" $@
1122	fi
1123	if [ $DRYRUN = 0 ]; then
1124	if ( [ ! -d ${1} ] && [ ! -f ${1} ] ) ; then
1125	echo "WARNING : IGCM_sys_Get_Master ${1} DOES NOT EXIST ."
1126	IGCM_debug_PopStack "IGCM_sys_Get_Master"
1127	return
1128	fi
1129
1130	typeset NB_ESSAI DELAI status i
1131	# number of tentative
1132	NB_ESSAI=3
1133	# time delay between tentative
1134	DELAI=2
1135
1136	i=0
1137	while [ $i -lt $NB_ESSAI ] ; do
1138	\cp -urL $1 $2 > /tmp/out_command.$$ 2>&1
1139	status=$?
1140	if [ ${status} -gt 0 ]; then
1141	IGCM_debug_Print 2 "IGCM_sys_Get_Master : cp failed error code ${status} ${i}/${NB_ESSAI}"
1142	IGCM_debug_Print 2 "IGCM_sys_Get_Master : sleep ${DELAI} seconds and try again."
1143	sleep $DELAI
1144	else
1145	break
1146	fi
1147	(( i = i + 1 ))
1148	done
1149
1150	if [ ${status} -gt 0 ] ; then
1151	echo "IGCM_sys_Get_Master : error."
1152	cat /tmp/out_command.$$
1153	IGCM_debug_Exit "IGCM_sys_Get_Master"
1154	else
1155	\rm /tmp/out_command.$$
1156	fi
1157	fi
1158	IGCM_debug_PopStack "IGCM_sys_Get_Master"
1159	}
1160
1161	#====================================================
1162	#- Call IGCM_sys_Mirror_libIGCM now !
1163	if ( $MirrorlibIGCM ) ; then
1164	IGCM_sys_Mirror_libIGCM
1165	fi
1166
1167	#D-#==================================================
1168	#D-function IGCM_sys_Put_Rest
1169	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
1170	#D- File and target directory must exist.
1171	#D-* Examples:
1172	#D-
1173	function IGCM_sys_Put_Rest {
1174	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
1175	if ( $DEBUG_sys ) ; then
1176	echo "IGCM_sys_Put_Rest :" $@
1177	fi
1178	if [ $DRYRUN = 0 ]; then
1179	if [ ! -f ${1} ] ; then
1180	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
1181	IGCM_debug_Exit "IGCM_sys_Put_Rest"
1182	fi
1183
1184	typeset status
1185	#
1186	# USUAL WAY
1187	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1188	status=$?
1189
1190	# #RSYNC WITH NETWORK SSH CALL
1191	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
1192	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
1193
1194	# #RSYNC WITH NFS USE
1195	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
1196	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
1197
1198	# status=$?
1199	# IGCM_sys_Rsync_out $status
1200
1201	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1202	# (( status=status+$? ))
1203
1204	if [ ${status} -gt 0 ] ; then
1205	echo "IGCM_sys_Put_Rest : cp failed error code ${status}"
1206	[ -f ${1} ] && ls -l ${1}
1207	[ -f ${2} ] && ls -l ${2}
1208	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1209	cat /tmp/out_command.$$
1210	IGCM_debug_Exit "IGCM_sys_Put_Rest"
1211	else
1212
1213	if [ X${JobType} = XRUN ] ; then
1214	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1215	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1216	fi
1217
1218	\rm /tmp/out_command.$$
1219	fi
1220	fi
1221	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
1222	}
1223
1224	#D-#==================================================
1225	#D-function IGCM_sys_PutBuffer_Rest
1226	#D-* Purpose: Put computied restarts on ${SCRATCHDIR}.
1227	#D- File and target directory must exist.
1228	#D-* Examples:
1229	#D-
1230	function IGCM_sys_PutBuffer_Rest {
1231	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Rest" $@
1232	if ( $DEBUG_sys ) ; then
1233	echo "IGCM_sys_PutBuffer_Rest :" $@
1234	fi
1235	if [ $DRYRUN = 0 ]; then
1236	if [ ! -f ${1} ] ; then
1237	echo "ERROR : IGCM_sys_PutBuffer_Rest ${1} DOES NOT EXIST ."
1238	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
1239	fi
1240
1241	typeset status
1242	#
1243	# USUAL WAY
1244	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1245	status=$?
1246
1247	if [ ${status} -gt 0 ] ; then
1248	echo "IGCM_sys_PutBuffer_Rest : error code ${status}"
1249	[ -f ${2} ] && ls -l ${2}
1250	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1251	cat /tmp/out_command.$$
1252	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
1253	else
1254
1255	if [ X${JobType} = XRUN ] ; then
1256	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1257	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1258	fi
1259
1260	\rm /tmp/out_command.$$
1261	fi
1262	fi
1263	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Rest"
1264	}
1265
1266	#D-#==================================================
1267	#D-function IGCM_sys_PrepareTaredRestart
1268	#D-* Purpose: Prepare tared restart to be access by computing job. Identity here.
1269	#D-* Examples:
1270	#D-
1271	function IGCM_sys_PrepareTaredRestart {
1272	IGCM_debug_PushStack "IGCM_sys_PrepareTaredRestart" $@
1273	if [ $DRYRUN = 0 ]; then
1274	[ ! -f $( basename $1 ) ] && IGCM_sys_Get $1 .
1275	fi
1276	IGCM_debug_PopStack "IGCM_sys_PrepareTaredRestart"
1277	}
1278
1279	#D-#==================================================
1280	#D-function IGCM_sys_Put_Out
1281	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
1282	#D-* Examples:
1283	#D-
1284	function IGCM_sys_Put_Out {
1285	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
1286	if ( $DEBUG_sys ) ; then
1287	echo "IGCM_sys_Put_Out :" $@
1288	fi
1289
1290	typeset NB_ESSAI DELAI status i exist skip
1291
1292	# number of tentative
1293	NB_ESSAI=3
1294	# time delay between tentative
1295	DELAI=2
1296
1297	if [ $DRYRUN = 0 ]; then
1298	if [ ! -f ${1} ] ; then
1299	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
1300	IGCM_debug_PopStack "IGCM_sys_Put_Out"
1301	return 1
1302	fi
1303	#
1304	IGCM_sys_MkdirArchive $( dirname $2 )
1305	#
1306
1307	#=====================================================
1308	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
1309	#=====================================================
1310
1311	#echo ${2} \| grep "${R_OUT}" > /dev/null 2>&1
1312	#if [ $? -eq 0 ] ; then
1313	# typeset WORKPATH FILEPATH
1314	# WORKPATH=$( dirname $2 \| sed -e "s\|${R_OUT}\|${R_BUF}\|" )
1315	# IGCM_sys_MkdirWork ${WORKPATH}
1316	# FILEPATH=${WORKPATH}/$( basename $2 )
1317	# #
1318	# IGCM_sys_Cp ${1} ${FILEPATH}
1319	#fi
1320
1321	exist=false
1322	skip=false
1323	if [ -f $2 ] ; then
1324	IGCM_debug_Print 1 "$2 already exist"
1325	ccc_hsm get $2
1326	exist=true
1327	if [ "X$( diff $1 $2 )" = X ] ; then
1328	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
1329	skip=true
1330	else
1331	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
1332	skip=false
1333	fi
1334	fi
1335	#
1336	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
1337	IGCM_sys_Chmod u+w $2
1338	fi
1339
1340	if [ X${skip} = Xfalse ] ; then
1341	i=0
1342	while [ $i -lt $NB_ESSAI ] ; do
1343	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
1344	# USUAL WAY
1345	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1346	status=$?
1347	else
1348	# NOT SO USUAL WAY
1349	\mv $1 $2 > /tmp/out_command.$$ 2>&1
1350	status=$?
1351	fi
1352	if [ ${status} -gt 0 ]; then
1353	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
1354	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
1355	[ -f ${1} ] && ls -l ${1}
1356	[ -f ${2} ] && ls -l ${2}
1357	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1358	sleep $DELAI
1359	else
1360	break
1361	fi
1362	(( i = i + 1 ))
1363	done
1364	fi
1365
1366	# #RSYNC WITH NETWORK SSH CALL
1367	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
1368	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
1369
1370	# #RSYNC WITH NFS USE
1371	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
1372	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
1373
1374	# status=$?
1375	# IGCM_sys_Rsync_out $status
1376
1377	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1378	# (( status=status+$? ))
1379
1380	if [ ${status} -gt 0 ] ; then
1381	echo "IGCM_sys_Put_Out : error."
1382	[ -f ${1} ] && ls -l ${1}
1383	[ -f ${2} ] && ls -l ${2}
1384	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1385	cat /tmp/out_command.$$
1386	IGCM_debug_Exit "IGCM_sys_Put_Out"
1387	else
1388
1389	if [ X${JobType} = XRUN ] ; then
1390	if [ X${3} = X ] ; then
1391	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1392	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1393	fi
1394	fi
1395
1396	\rm /tmp/out_command.$$
1397	fi
1398	fi
1399	IGCM_debug_PopStack "IGCM_sys_Put_Out"
1400	return 0
1401	}
1402
1403	#D-#==================================================
1404	#D-function IGCM_sys_PutBuffer_Out
1405	#D-* Purpose: Copy a file on ${SCRATCHDIR} after having chmod it in readonly
1406	#D-* Examples:
1407	#D-
1408	function IGCM_sys_PutBuffer_Out {
1409	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Out" $@
1410	if ( $DEBUG_sys ) ; then
1411	echo "IGCM_sys_PutBuffer_Out :" $@
1412	fi
1413
1414	typeset NB_ESSAI DELAI status i exist skip
1415
1416	# number of tentative
1417	NB_ESSAI=3
1418	# time delay between tentative
1419	DELAI=2
1420
1421	if [ $DRYRUN = 0 ]; then
1422	if [ ! -f ${1} ] ; then
1423	echo "WARNING : IGCM_sys_PutBuffer_Out ${1} DOES NOT EXIST ."
1424	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
1425	return 1
1426	fi
1427	#
1428	IGCM_sys_Mkdir $( dirname $2 )
1429	#
1430
1431	exist=false
1432	skip=false
1433	if [ -f $2 ] ; then
1434	IGCM_debug_Print 1 "$2 already exist"
1435	exist=true
1436	if [ "X$( diff $1 $2 )" = X ] ; then
1437	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
1438	status=0
1439	skip=true
1440	else
1441	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
1442	skip=false
1443	fi
1444	fi
1445	#
1446	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
1447	IGCM_sys_Chmod u+w $2
1448	fi
1449
1450	if [ X${skip} = Xfalse ] ; then
1451	i=0
1452	while [ $i -lt $NB_ESSAI ] ; do
1453	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
1454	# USUAL WAY
1455	\cp $1 $2 > /tmp/out_command.$$ 2>&1
1456	status=$?
1457	else
1458	# NOT SO USUAL WAY
1459	\mv $1 $2 > /tmp/out_command.$$ 2>&1
1460	status=$?
1461	fi
1462	if [ ${status} -gt 0 ]; then
1463	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
1464	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : sleep ${DELAI} seconds and try again."
1465	[ -f ${2} ] && ls -l ${2}
1466	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1467	sleep $DELAI
1468	else
1469	break
1470	fi
1471	(( i = i + 1 ))
1472	done
1473	fi
1474
1475	if [ ${status} -gt 0 ] ; then
1476	echo "IGCM_sys_PutBuffer_Out : error."
1477	[ -f ${2} ] && ls -l ${2}
1478	[ -f ${2}/${1} ] && ls -l ${2}/${1}
1479	cat /tmp/out_command.$$
1480	IGCM_debug_Exit "IGCM_sys_PutBuffer_Out"
1481	else
1482
1483	if [ X${JobType} = XRUN ] ; then
1484	if [ X${3} = X ] ; then
1485	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
1486	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
1487	fi
1488	fi
1489
1490	\rm /tmp/out_command.$$
1491	fi
1492	fi
1493	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
1494	return 0
1495	}
1496
1497	#D-#==================================================
1498	#D-function IGCM_sys_Get
1499	#D-* Purpose: Get a file from ${ARCHIVE}
1500	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
1501	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
1502	function IGCM_sys_Get {
1503	IGCM_debug_PushStack "IGCM_sys_Get" $@
1504
1505	typeset DEST dm_liste target file_work
1506	typeset NB_ESSAI DELAI status i
1507
1508	if ( $DEBUG_sys ) ; then
1509	echo "IGCM_sys_Get :" $@
1510	fi
1511
1512	# number of tentative
1513	NB_ESSAI=3
1514	# time delay between tentative
1515	DELAI=2
1516
1517	if [ $DRYRUN -le 2 ]; then
1518	if [ X${1} = X'/l' ] ; then
1519	eval set +A dm_liste \${${2}}
1520	else
1521	eval set +A dm_liste ${1}
1522	fi
1523	eval DEST=\${${#}}
1524
1525	#=====================================================
1526	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
1527	#=====================================================
1528
1529	# Is it an R_OUT file (not R_IN) ?
1530	#echo ${dm_liste[0]} \| grep "${R_OUT}" > /dev/null 2>&1
1531	#if [ $? -eq 0 ] ; then
1532	# # Yes ? then we try to get it in SCRATCHDIR
1533	# set +A file_work $( echo ${dm_liste[*]} \| sed -e "s\|${R_OUT}\|${R_BUF}\|g" )
1534	# if [ -f ${file_work[0]} ] ; then
1535	# IGCM_sys_Cp ${file_work[*]} ${DEST}
1536	# IGCM_debug_PopStack "IGCM_sys_Get"
1537	# return
1538	# fi
1539	#fi
1540
1541	ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1542	status=$?
1543	if [ ${status} -gt 0 ] ; then
1544	echo "WARNING IGCM_sys_Get : error code ${status}"
1545	cat /tmp/out_command.$$
1546	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
1547	fi
1548
1549	#if [ ${status} -gt 0 ] ; then
1550	# if [ ! "X$( grep "Lost dmusrcmd connection" /tmp/out_command.$$ )" = "X" ] ; then
1551	# cat /tmp/out_command.$$
1552	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
1553	# sleep 30
1554	# echo "We try another time"
1555	## dmget ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1556	# ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
1557	# status=$?
1558	# if [ ${status} -gt 0 ] ; then
1559	# echo "ERROR IGCM_sys_Get : again demigration error :"
1560	# cat /tmp/out_command.$$
1561	# IGCM_debug_Exit "IGCM_sys_Get"
1562	# fi
1563	# else
1564	# echo "ERROR IGCM_sys_Get : demigration error :"
1565	# cat /tmp/out_command.$$
1566	# IGCM_debug_Exit "IGCM_sys_Get"
1567	# fi
1568	#fi
1569
1570	# #RSYNC WITH NETWORK SSH CALL
1571	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
1572	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
1573
1574	# #RSYNC WITH NFS USE
1575	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
1576	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
1577
1578	# status=$?
1579	# IGCM_sys_Rsync_out $status
1580
1581	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
1582	# (( status=status+$? ))
1583
1584	#USUAL WAY
1585	if [ X${1} = X'/l' ] ; then
1586	for target in ${dm_liste[*]} ; do
1587	local_file=$( basename ${target} )
1588	# test if the target file is present before the loop
1589	IGCM_sys_TestFileArchive ${target}
1590	status=$?
1591	if [ ${status} -gt 0 ] ; then
1592	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
1593	IGCM_debug_Exit "IGCM_sys_Get"
1594	else
1595	i=0
1596	while [ $i -lt $NB_ESSAI ] ; do
1597	#if [ X${DoLink} = Xtrue ] ; then
1598	# \ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1599	# status=$?
1600	# else
1601	# \cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1602	# status=$?
1603	#fi
1604	\ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1605	status=$?
1606	if [ ${status} -gt 0 ]; then
1607	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
1608	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
1609	sleep $DELAI
1610	else
1611	break
1612	fi
1613	(( i = i + 1 ))
1614	done
1615	if [ ${status} -gt 0 ] ; then
1616	echo "IGCM_sys_Get : error"
1617	cat /tmp/out_command.$$
1618	\rm /tmp/out_command.$$
1619	IGCM_debug_Exit "IGCM_sys_Get"
1620	else
1621	\rm /tmp/out_command.$$
1622	fi
1623	fi
1624	done
1625	else
1626	i=0
1627	while [ $i -lt $NB_ESSAI ] ; do
1628	\cp ${dm_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
1629	status=$?
1630	if [ ${status} -gt 0 ]; then
1631	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
1632	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
1633	sleep $DELAI
1634	else
1635	break
1636	fi
1637	(( i = i + 1 ))
1638	done
1639	if [ ${status} -gt 0 ] ; then
1640	echo "IGCM_sys_Get : error"
1641	cat /tmp/out_command.$$
1642	\rm /tmp/out_command.$$
1643	IGCM_debug_Exit "IGCM_sys_Get"
1644	else
1645	\rm /tmp/out_command.$$
1646	fi
1647	fi
1648	fi
1649	IGCM_debug_PopStack "IGCM_sys_Get"
1650	}
1651
1652	#D-#==================================================
1653	#D-function IGCM_sys_GetBuffer
1654	#D-* Purpose: Get a file from ${SCRATCHDIR}
1655	#D-* Examples: IGCM_sys_GetBuffer myfile /destpath/myfile_with_PREFIX
1656	#D- IGCM_sys_GetBuffer /l Array_contain_myfiles /destpath/
1657	function IGCM_sys_GetBuffer {
1658	IGCM_debug_PushStack "IGCM_sys_GetBuffer" $@
1659
1660	typeset DEST buf_liste target file_work
1661	typeset NB_ESSAI DELAI status i
1662
1663	if ( $DEBUG_sys ) ; then
1664	echo "IGCM_sys_GetBuffer :" $@
1665	fi
1666
1667	# number of tentative
1668	NB_ESSAI=3
1669	# time delay between tentative
1670	DELAI=2
1671
1672	if [ $DRYRUN -le 2 ]; then
1673	if [ X${1} = X'/l' ] ; then
1674	eval set +A buf_liste \${${2}}
1675	else
1676	eval set +A buf_liste ${1}
1677	fi
1678	eval DEST=\${${#}}
1679
1680	#USUAL WAY
1681	if [ X${1} = X'/l' ] ; then
1682	for target in ${buf_liste[*]} ; do
1683	local_file=$( basename ${target} )
1684	i=0
1685	while [ $i -lt $NB_ESSAI ] ; do
1686	\cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
1687	status=$?
1688	if [ ${status} -gt 0 ]; then
1689	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
1690	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
1691	sleep $DELAI
1692	else
1693	break
1694	fi
1695	(( i = i + 1 ))
1696	done
1697	if [ ${status} -gt 0 ] ; then
1698	echo "IGCM_sys_Get : error"
1699	cat /tmp/out_command.$$
1700	\rm /tmp/out_command.$$
1701	IGCM_debug_Exit "IGCM_sys_GetBuffer"
1702	else
1703	\rm /tmp/out_command.$$
1704	fi
1705	done
1706	else
1707	i=0
1708	while [ $i -lt $NB_ESSAI ] ; do
1709	\cp ${buf_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
1710	status=$?
1711	if [ ${status} -gt 0 ]; then
1712	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
1713	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
1714	sleep $DELAI
1715	else
1716	break
1717	fi
1718	(( i = i + 1 ))
1719	done
1720	if [ ${status} -gt 0 ] ; then
1721	echo "IGCM_sys_Get : error"
1722	cat /tmp/out_command.$$
1723	\rm /tmp/out_command.$$
1724	IGCM_debug_Exit "IGCM_sys_GetBuffer"
1725	else
1726	\rm /tmp/out_command.$$
1727	fi
1728	fi
1729	fi
1730	IGCM_debug_PopStack "IGCM_sys_GetBuffer"
1731	}
1732
1733	#D-#==================================================
1734	#D-function IGCM_sys_GetDate_FichWork
1735	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem WORK
1736	#D-* Examples:
1737	#D-
1738	function IGCM_sys_GetDate_FichWork {
1739	IGCM_debug_PushStack "IGCM_sys_FichWork" $@
1740	if ( $DEBUG_sys ) ; then
1741	echo "IGCM_sys_GetDate_FichWork :" $@
1742	fi
1743
1744	if [ $# -ge 3 ] ; then
1745	mode=$3
1746	TimeStyle=$4
1747	else
1748	mode="default"
1749	TimeStyle="%Y%m%d%H%M%S"
1750	fi
1751
1752	typeset dateF
1753	set +A dateF -- $( ls -l --full-time --time-style=+"${TimeStyle}" ${1} )
1754
1755	case $mode in
1756	"default")
1757	eval ${2}=${dateF[5]}
1758	;;
1759	"SplitFields")
1760	eval ${2}="${dateF[5]}\ ${dateF[6]}"
1761	;;
1762	esac
1763
1764
1765	# donne la date filesys d'un fichier sur la machine work
1766	IGCM_debug_PopStack "IGCM_sys_FichWork"
1767	}
1768
1769	#D-#==================================================
1770	#D-function IGCM_sys_GetDate_FichArchive
1771	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem ARCHIVE
1772	#D-* Examples:
1773	#D-
1774	function IGCM_sys_GetDate_FichArchive {
1775	IGCM_debug_PushStack "IGCM_sys_FichArchive" $@
1776	if ( $DEBUG_sys ) ; then
1777	echo "IGCM_sys_GetDate_FichArchive :" $@
1778	fi
1779	typeset dateF
1780	set +A dateF -- $( ls -l --full-time --time-style='+%Y%m%d%H%M%S' ${1} )
1781	eval ${2}=${dateF[5]}
1782
1783	IGCM_debug_PopStack "IGCM_sys_FichArchive"
1784	}
1785
1786	#D-#==================================================
1787	#D-function IGCM_sys_GetDate_Monitoring
1788	#D-* Purpose: get the last year for which the monitoring has been computed
1789	#D-* Examples:
1790	#D-
1791	function IGCM_sys_GetDate_Monitoring {
1792	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
1793	if ( $DEBUG_sys ) ; then
1794	echo "IGCM_sys_GetDate_Monitoring :" $@
1795	fi
1796
1797	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
1798
1799	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
1800	}
1801
1802	#D-#==================================================
1803	#D-function IGCM_sys_Dods_Rm
1804	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
1805	#D-* Examples:
1806	#D-
1807	function IGCM_sys_Dods_Rm {
1808	if ( $DEBUG_sys ) ; then
1809	echo "IGCM_sys_Dods_Rm :" $@
1810	fi
1811	typeset status
1812	if [ $DRYRUN = 0 ]; then
1813
1814	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
1815	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
1816	# echo "Nothing has been done."
1817	# return
1818	# fi
1819
1820	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
1821	status=$?
1822
1823	# if [ ${status} -gt 0 ] ; then
1824	# echo "IGCM_sys_Dods_Rm : error."
1825	# cat out_dods_rm
1826	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
1827	# else
1828	# rm out_dods_rm
1829	# fi
1830
1831	fi
1832	return $status
1833	}
1834
1835	#D-#==================================================
1836	#D-function IGCM_sys_Dods_Cp
1837	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
1838	#D-* Examples:
1839	#D-
1840	function IGCM_sys_Dods_Cp {
1841	if ( $DEBUG_sys ) ; then
1842	echo "IGCM_sys_Dods_Cp :" $@
1843	fi
1844	typeset status
1845	if [ $DRYRUN = 0 ]; then
1846
1847	# if [ ! -d ${R_SAVE}/${1} ] ; then
1848	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
1849	# echo "Nothing has been done."
1850	# return
1851	# fi
1852
1853	/ccc/cont003/home/dsm/p86ipsl/bin/dods_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
1854	status=$?
1855
1856	# if [ ${status} -gt 0 ] ; then
1857	# echo "IGCM_sys_Dods_Cp : error."
1858	# cat out_dods_cp
1859	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1860	# else
1861	# rm out_dods_cp
1862	# fi
1863
1864	fi
1865	return $status
1866	}
1867
1868	#D-#==================================================
1869	#D-function IGCM_sys_Put_Dods
1870	#D-* Purpose: Put $(ARCHIVE) files on DODS internet protocole.
1871	#D-* Examples:
1872	#D-
1873	function IGCM_sys_Put_Dods {
1874	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1875	if ( $DEBUG_sys ) ; then
1876	echo "IGCM_sys_Put_Dods :" $@
1877	fi
1878	#set -vx
1879	typeset status
1880	if [ $DRYRUN = 0 ]; then
1881	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1882	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1883	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1884	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1885	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1886	return
1887	fi
1888
1889	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1890	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1891	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1892	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1893	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1894	return
1895	fi
1896	#
1897	if [ -d ${R_SAVE}/${1} ] ; then
1898	cd ${R_SAVE}
1899	elif [ -d ${R_FIGR}/${1} ] ; then
1900	cd ${R_FIGR}
1901	fi
1902
1903	IGCM_sys_Dods_Rm ${1}
1904	IGCM_sys_Dods_Cp ${1}
1905	status=0
1906
1907	if [ ${status} -gt 0 ] ; then
1908	echo "IGCM_sys_Put_Dods : error."
1909	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1910	fi
1911	fi
1912	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1913	}
1914
1915	##############################################################
1916	# REBUILD OPERATOR
1917
1918	#D-#==================================================
1919	#D-function IGCM_sys_rebuild
1920	#D-* Purpose: rebuild parallel files
1921	#D-* Examples:
1922	#D-
1923	function IGCM_sys_rebuild {
1924	IGCM_debug_PushStack "IGCM_sys_rebuild" $@
1925	if ( $DEBUG_sys ) ; then
1926	echo "IGCM_sys_rebuild :" $@
1927	fi
1928
1929	typeset NB_ESSAI DELAI status i firstArg
1930	# number of tentative
1931	NB_ESSAI=3
1932	# time delay between tentative
1933	DELAI=2
1934
1935	i=0
1936	while [ $i -lt $NB_ESSAI ] ; do
1937	~p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -f -o $@ > /tmp/out_command.$$ 2>&1
1938	status=$?
1939	if [ ${status} -gt 0 ] ; then
1940	IGCM_debug_Print 2 "IGCM_sys_rebuild : error code ${status}"
1941	cat /tmp/out_command.$$
1942	\rm /tmp/out_command.$$
1943	IGCM_debug_Print 2 "IGCM_sys_rebuild : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
1944	firstArg=${1}
1945	\rm ${firstArg}
1946	sleep $DELAI
1947	else
1948	\rm /tmp/out_command.$$
1949	break
1950	fi
1951	(( i = i + 1 ))
1952	done
1953
1954	if [ ${status} -gt 0 ] ; then
1955	echo "IGCM_sys_rebuild : rebuild error code is ${status}"
1956	IGCM_debug_Exit "rebuild"
1957	fi
1958
1959	IGCM_debug_PopStack "IGCM_sys_rebuild"
1960	}
1961
1962	#D-#==================================================
1963	#D-function IGCM_sys_rebuild_station
1964	#D-* Purpose: rebuild parallel files describing station
1965	#D-* Examples:
1966	#D-
1967	function IGCM_sys_rebuild_station {
1968	IGCM_debug_PushStack "IGCM_sys_rebuild_station" $@
1969	typeset i list_opt file_in file_out prefix_invert list_invert
1970	if ( $DEBUG_sys ) ; then
1971	echo "IGCM_sys_rebuild_station :" $@
1972	fi
1973	list_opt=$@
1974
1975	# Invert Axis : t,x -> x,t
1976	# t,pres,x -> x,t,pres
1977	# So that we can concatenate along x
1978	i=0
1979	for file_in in ${list_opt} ; do
1980	(( i = i + 1))
1981	[ ${i} = 1 ] && file_out=${file_in} && continue
1982	# detect time counter and do the job only if present
1983	var_unlim=$(ncdump -h ${file_in} \| grep UNLIMITED \| cut -d ' ' -f 1 \| sed -e 's/^[ \t]//' -e 's/[ \t]$//')
1984	if [ X${var_unlim} = Xtime_counter ] ; then
1985	prefix_invert=$( basename ${file_in} .nc )
1986	IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs ${file_in} ${prefix_invert}_xt.nc
1987	list_invert[${#list_invert[*]}]=${prefix_invert}_xt.nc
1988	fi
1989	done
1990
1991	# Concatenate
1992	IGCM_sys_ncrcat ${list_invert[*]} histstn_xt.nc
1993
1994	# Re-ivert file
1995	IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x histstn_xt.nc ${file_out}
1996
1997	# Station re-ordering is too expansive to be run within libIGCM
1998	# This is due to (ncpdq - nrcat - ncpdq) I/O sequence.
1999	# This re-ordering must be done "in memory" by the cmorization process
2000	# Anyway this is the best sequence using (ncpdq - nrcat - ncpdq)
2001	# BEGIN reordering
2002
2003	# Only LMDZ text output contains the exact ordering of the station.
2004	# We isolate this in the code below:
2005	# 0 38 -157.5000000000000 70.98591549295774
2006	# 0 54 27.49999999999999 67.18309859154928
2007	# 0 56 -62.50000000000001 82.39436619718309
2008	# 0 79 12.49999999999999 78.59154929577466
2009	# 0 116 -165.0000000000000 76.05633802816901
2010	# 0 117 130.0000000000000 70.98591549295774
2011	# 0 118 110.0000000000000 87.46478873239437
2012	# 1 40 4.999999999999995 51.97183098591550
2013
2014	# typeset iStation iProc list_opt file_in file_out prefix_invert
2015	# typeset -Z4 j4
2016	# typeset -Z3 j3
2017
2018	# unset list_opt
2019	# set +A list_opt $@
2020
2021	# Filename after rebuild
2022	# file_out=${list_opt[0]}
2023	# Prefix of output files
2024	# prefix_invert=$( basename ${file_out} .nc )
2025	# Number of procs
2026	# num_proc=$( grep -i mpi_size ${PREFIX}_${Exe_Output} \| wc -l )
2027
2028	# iProc=0
2029	# while [ ${iProc} -lt ${num_proc} ] ; do
2030	# Array containing Station as a number
2031	# unset proc_stn
2032	# set +A proc_stn $( grep "iophy_mpi rank ip lon lat $iProc" ${PREFIX}_${Exe_Output} \| sed -e "s/iophy_mpi rank ip lon lat //g" \| gawk ' {print $2}' )
2033	# Number of stations produced by processor proc
2034	# stationLast=${#proc_stn[*]}
2035	# Proc number on 4 digits
2036	# j4=${iProc}
2037	# Init
2038	# iStation=0
2039	# while [ ${iStation} -lt ${stationLast} ] ; do
2040	# Station number on 3 digits
2041	# j3=${proc_stn[${iStation}]}
2042	# Extract station
2043	# Invert Axis : t,x -> x,t
2044	# t,pres,x -> x,t,pres
2045	# So that we can concatenate along x
2046	# IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs -d x,$iStation,$iStation ${prefix_invert}_${j4}.nc ${prefix_invert}_stn_${j3}.nc
2047	# (( iStation = iStation + 1 ))
2048	# done
2049	# (( iProc = iProc + 1 ))
2050	# done
2051
2052	# Concatenate all station along x
2053	# IGCM_sys_ncrcat ${prefix_invert}_stn_???.nc ${prefix_invert}_xt.nc
2054
2055	# Re-invert file
2056	# IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x ${prefix_invert}_xt.nc ${file_out}
2057
2058	# END reordering
2059
2060	IGCM_debug_PopStack "IGCM_sys_rebuild_station"
2061	}
2062
2063	############################################################
2064	# Activate Running Environnment Variables
2065
2066	#D-#==================================================
2067	#D-function IGCM_sys_desactiv_variables
2068	#D-* Purpose: set environement variables prior to execution
2069	#D-* Examples:
2070	#D-
2071	function IGCM_sys_activ_variables {
2072	IGCM_debug_PushStack "IGCM_sys_activ_variables"
2073	if ( $DEBUG_sys ) ; then
2074	echo "IGCM_sys_activ_variables"
2075	fi
2076
2077	# --------------------------------------------------------------------
2078	#D- MPI specifications
2079	# --------------------------------------------------------------------
2080
2081	# --------------------------------------------------------------------
2082	#D- Other specifications
2083	# --------------------------------------------------------------------
2084
2085	ulimit -s unlimited
2086
2087	IGCM_debug_PopStack "IGCM_sys_activ_variables"
2088	}
2089
2090	############################################################
2091	# Desactivate Running Environnment Variables
2092
2093	#D-#==================================================
2094	#D-function IGCM_sys_desactiv_variables
2095	#D-* Purpose: unset environement variables after execution
2096	#D-* Examples:
2097	#D-
2098	function IGCM_sys_desactiv_variables {
2099	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
2100	if ( $DEBUG_sys ) ; then
2101	echo "IGCM_sys_desactiv_variables"
2102	fi
2103	# --------------------------------------------------------------------
2104	#D- MPI specifications
2105	# --------------------------------------------------------------------
2106
2107	# --------------------------------------------------------------------
2108	#D- Other specifications
2109	# --------------------------------------------------------------------
2110
2111	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
2112	}
2113
2114	############################################################
2115	# Build MPI/OMP scripts run file (dummy function)
2116
2117	#D-#==================================================
2118	#D-function IGCM_sys_build_run_file
2119	#D-* Purpose: build run file (deprecated)
2120	#D-* Examples:
2121	#D-
2122	function IGCM_sys_build_run_file {
2123
2124	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
2125
2126	}
2127
2128	############################################################
2129	# Build MPI/OMP scripts
2130
2131	#D-#==================================================
2132	#D-function IGCM_sys_build_execution_scripts
2133	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
2134	#D-* Examples:
2135	#D-
2136	function IGCM_sys_build_execution_scripts
2137	{
2138	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
2139	if ( $DEBUG_sys ) ; then
2140	echo "IGCM_sys_build_execution_scripts " $@
2141	fi
2142
2143	EXECUTION=${HOST_MPIRUN_COMMAND}
2144
2145	# MPMD mode
2146	if ( ${OK_PARA_MPMD} ) ; then
2147
2148	# Only MPI (MPMD)
2149	if ( ! ${OK_PARA_OMP} ) ; then
2150
2151	if [ -f run_file ] ; then
2152	IGCM_sys_Rm -f run_file
2153	fi
2154	touch run_file
2155
2156	# Build run_file
2157
2158	# First loop on the components for the coupler ie oasis (only if oasis3)
2159	# the coupler ie oasis3 must be the first one
2160	for comp in ${config_ListOfComponents[*]} ; do
2161
2162	eval ExeNameIn=\${config_Executable_${comp}[0]}
2163	eval ExeNameOut=\${config_Executable_${comp}[1]}
2164
2165	# for CPL component only
2166	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
2167
2168	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2169	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2170	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
2171	fi
2172	done
2173
2174	# Then second loop on the components
2175	for comp in ${config_ListOfComponents[*]} ; do
2176
2177	eval ExeNameIn=\${config_Executable_${comp}[0]}
2178	eval ExeNameOut=\${config_Executable_${comp}[1]}
2179
2180	# Only if we really have an executable for the component and not the coupler ie oasis:
2181	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
2182
2183	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2184	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2185	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
2186	fi
2187	done
2188
2189	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
2190
2191	IGCM_sys_Chmod u+x run_file
2192	if ( $DEBUG_sys ) ; then
2193	echo "run_file contains : "
2194	cat run_file
2195	fi
2196
2197	# MPI-OpenMP (MPMD)
2198	else
2199
2200	# Use of mpirun instead of ccc_mprun
2201	EXECUTION="time mpirun"
2202
2203	# Hosts treatment
2204	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
2205
2206	i=0
2207	rm -f hosts rankfile
2208	IGCM_debug_Print 1 "sys Curie, Hosts available :"
2209	for nodes in `cat hosts.tmp`
2210	do
2211	host[$i]=$nodes
2212	echo "${host[$i]}" >> hosts
2213	IGCM_debug_Print 1 ${host[$i]}
2214	i=$((i+1))
2215	done
2216	rm -f hosts.tmp
2217
2218	listnodes=${host[*]}
2219
2220	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
2221
2222	# Initialisation
2223	rank=0
2224	current_core=0
2225	core_per_node=16
2226	init_exec=n
2227
2228	# Loop on the components
2229	for comp in ${config_ListOfComponents[*]} ; do
2230
2231	eval ExeNameIn=\${config_Executable_${comp}[0]}
2232	eval ExeNameOut=\${config_Executable_${comp}[1]}
2233
2234	# Not possible if oasis has an executable (i.e old version of oasis3)
2235	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
2236	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
2237	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
2238	IGCM_debug_Verif_Exit
2239	fi
2240
2241	# Only if we really have an executable for the component :
2242	if [ "X${ExeNameOut}" != X\"\" ] ; then
2243
2244	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2245	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2246
2247	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
2248	echo "" >> script_${ExeNameOut}.ksh
2249	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
2250
2251	# Check if the number of threads is correct
2252	case ${comp_proc_omp_loc} in
2253	2\|4\|8\|16)
2254	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
2255	;;
2256	*)
2257	IGCM_debug_Exit "ERROR with OMP parameters !"
2258	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
2259	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
2260	IGCM_debug_Verif_Exit
2261	;;
2262	esac
2263	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
2264	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
2265	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
2266	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
2267	fi
2268	#echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
2269	#echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
2270	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2271	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
2272
2273	if [ ${init_exec} = y ] ; then
2274	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2275	else
2276	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2277	init_exec=y
2278	fi
2279
2280	# Build rankfile : method used to assign cores and nodes for the MPI process
2281	# Ex :
2282	#rank 0=curie5296 slot=0,1,2,3
2283	#rank 1=curie5296 slot=4,5,6,7
2284	# Example of final command :
2285	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
2286	# with script_lmdz.x.ksh :
2287	# #!/bin/ksh
2288	#export KMP_STACKSIZE=3g
2289	#export KMP_LIBRARY=turnaround
2290	#export MKL_SERIAL=YES
2291	#OMP_NUM_THREADS=4
2292	#./lmdz.x
2293	#
2294
2295	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
2296	(( index_host = current_core / core_per_node ))
2297	host_value=${host[${index_host}]}
2298	(( slot = current_core % core_per_node ))
2299	virg=","
2300	string_final=""
2301	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
2302	string=$index$virg
2303	string_final=$string_final$string
2304	done
2305	string_final=$( echo $string_final \| sed "s/.$//" )
2306	echo "rank $rank=$host_value slot=$string_final" >> rankfile
2307	(( rank = rank + 1 ))
2308	(( current_core = current_core + comp_proc_omp_loc ))
2309	done
2310	fi
2311
2312	done
2313	fi
2314
2315	# Only one executable (SPMD mode).
2316	else
2317
2318	for comp in ${config_ListOfComponents[*]} ; do
2319
2320	# Only if we really have an executable for the component :
2321	eval ExeNameOut=\${config_Executable_${comp}[1]}
2322	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
2323
2324	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
2325	echo "" >> script_${ExeNameOut}.ksh
2326	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
2327
2328	if ( ${OK_PARA_OMP} ) ; then
2329	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
2330	echo "" >> script_${ExeNameOut}.ksh
2331	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
2332	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
2333	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
2334	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
2335	fi
2336
2337	if ( ${OK_PARA_MPI} ) ; then
2338	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
2339	# Default : ccc_mprun used if nb_proc gt 1
2340	# to have out/err per process on different files
2341	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
2342	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2343	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
2344	else
2345	# Default : ccc_mprun is NOT used if nb_proc eq 1
2346	# to have out/err per process on different files
2347	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
2348	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
2349	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
2350	fi
2351
2352	IGCM_debug_Print 1 "sys Curie : script_${ExeNameOut}.ksh contains"
2353	cat script_${ExeNameOut}.ksh
2354
2355	fi
2356
2357	done
2358
2359	fi
2360
2361	IGCM_debug_Print 1 "sys Curie : execution command is "
2362	IGCM_debug_Print 1 "$EXECUTION"
2363
2364	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
2365	}
2366
2367	#D-#==================================================
2368	#D-function IGCM_sys_check_path
2369	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
2370	#D-* do not point to an important use directory. Stop immediately in that case.
2371	#D-* Examples:
2372	#D-
2373	function IGCM_sys_check_path {
2374	IGCM_debug_PushStack "IGCM_sys_check_path"
2375	if ( $DEBUG_sys ) ; then
2376	echo "IGCM_sys_check_path"
2377	fi
2378
2379	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
2380	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
2381	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
2382	IGCM_debug_Exit "This will stop the job"
2383	fi
2384	IGCM_debug_PopStack "IGCM_sys_check_path"
2385	}
2386
2387	#D-#==================================================
2388	#D-function IGCM_sys_check_quota
2389	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
2390	#D-* Examples:
2391	#D-
2392	function IGCM_sys_check_quota {
2393	IGCM_debug_PushStack "IGCM_sys_check_quota"
2394	if ( $DEBUG_sys ) ; then
2395	echo "IGCM_sys_check_quota"
2396	fi
2397	# Limit of quota (in %)
2398	limit_quota=90
2399
2400	# Check of the volume
2401	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
2402	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
2403
2404	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
2405
2406	unit_avail=${volume_avail: -1}
2407	unit_quota=${volume_quota: -1}
2408
2409	if [ "${unit_quota}" = "*" ] ; then
2410	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
2411	IGCM_debug_Print 1 "More than 100% of your quota is used"
2412	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2413	IGCM_debug_Print 1 "You must have more than 10% available to run"
2414	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2415	IGCM_debug_Verif_Exit
2416	fi
2417
2418	temp_avail=${volume_avail%%${volume_avail: -1}*}
2419	temp_quota=${volume_quota%%${volume_quota: -1}*}
2420
2421	if [ ! ${unit_avail} = ${unit_quota} ] ; then
2422
2423	# Convertion
2424	if [ ${volume_avail: -1} = "T" ] ; then
2425	(( temp_avail = temp_avail * 1000000000000 ))
2426	elif [ ${volume_avail: -1} = "G" ] ; then
2427	(( temp_avail = temp_avail * 1000000000 ))
2428	elif [ ${volume_avail: -1} = "M" ] ; then
2429	(( temp_avail = temp_avail * 1000000 ))
2430	elif [ ${volume_avail: -1} = "k" ] ; then
2431	(( temp_avail = temp_avail * 1000 ))
2432	else
2433	(( temp_avail = volume_avail ))
2434	fi
2435	if [ ${volume_quota: -1} = "T" ] ; then
2436	(( temp_quota = temp_quota * 1000000000000 ))
2437	elif [ ${volume_quota: -1} = "G" ] ; then
2438	(( temp_quota = temp_quota * 1000000000 ))
2439	elif [ ${volume_quota: -1} = "M" ] ; then
2440	(( temp_quota = temp_quota * 1000000 ))
2441	elif [ ${volume_quota: -1} = "k" ] ; then
2442	(( temp_quota = temp_quota * 1000 ))
2443	else
2444	(( temp_quota = volume_quota ))
2445	fi
2446	fi
2447
2448	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
2449	# echo "volume ratio is " $quota_volume
2450
2451	if [ ${quota_volume} -ge ${limit_quota} ] ; then
2452	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
2453	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
2454	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2455	IGCM_debug_Print 1 "You must have more than 10% available to run"
2456	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2457	IGCM_debug_Verif_Exit
2458	fi
2459
2460	fi
2461
2462	# Check of the number of inodes
2463
2464	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
2465	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
2466
2467	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
2468
2469	unit_avail=${inode_avail: -1}
2470	unit_quota=${inode_quota: -1}
2471
2472	if [ "${unit_quota}" = "*" ] ; then
2473	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
2474	IGCM_debug_Print 1 "More than 100% of your quota is used"
2475	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2476	IGCM_debug_Print 1 "You must have more than 10% available to run"
2477	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2478	IGCM_debug_Verif_Exit
2479	fi
2480
2481	temp_avail=${inode_avail%%${inode_avail: -1}*}
2482	temp_quota=${inode_quota%%${inode_quota: -1}*}
2483
2484	if [ ! ${unit_avail} = ${unit_quota} ] ; then
2485
2486	# Convertion
2487	if [ ${inode_avail: -1} = "T" ] ; then
2488	(( temp_avail = temp_avail * 1000000000000 ))
2489	elif [ ${inode_avail: -1} = "G" ] ; then
2490	(( temp_avail = temp_avail * 1000000000 ))
2491	elif [ ${inode_avail: -1} = "M" ] ; then
2492	(( temp_avail = temp_avail * 1000000 ))
2493	elif [ ${inode_avail: -1} = "k" ] ; then
2494	(( temp_avail = temp_avail * 1000 ))
2495	else
2496	(( temp_avail = inode_avail ))
2497	fi
2498
2499	if [ ${inode_quota: -1} = "T" ] ; then
2500	(( temp_quota = temp_quota * 1000000000000 ))
2501	elif [ ${inode_quota: -1} = "G" ] ; then
2502	(( temp_quota = temp_quota * 1000000000 ))
2503	elif [ ${inode_quota: -1} = "M" ] ; then
2504	(( temp_quota = temp_quota * 1000000 ))
2505	elif [ ${inode_quota: -1} = "k" ] ; then
2506	(( temp_quota = temp_quota * 1000 ))
2507	else
2508	(( temp_quota = inode_quota ))
2509	fi
2510	fi
2511	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
2512	# echo "inode ratio is " $quota_inode
2513
2514	if [ ${quota_inode} -ge ${limit_quota} ] ; then
2515	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
2516	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
2517	IGCM_debug_Print 1 "Use the ccc_quota command to check"
2518	IGCM_debug_Print 1 "You must have more than 10% available to run"
2519	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
2520	IGCM_debug_Verif_Exit
2521	fi
2522	fi
2523	IGCM_debug_PopStack "IGCM_sys_check_quota"
2524	}
2525
2526	#D-#==================================================
2527	#D-function IGCM_sys_GetJobID
2528	#D-* Purpose: Check if job_name is currently
2529	#D- running or in queue
2530	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
2531	#D-
2532	function IGCM_sys_GetJobID {
2533	IGCM_debug_PushStack "IGCM_sys_GetJobID"
2534	if ( $DEBUG_sys ) ; then
2535	echo "IGCM_sys_GetJobID"
2536	fi
2537
2538	# With -f option, the full job name is given in the last column
2539	ID=$( ccc_mstat -f -u $2 \| \
2540	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
2541
2542	eval ${3}=${ID}
2543
2544	IGCM_debug_PopStack "IGCM_sys_GetJobID"
2545	}
2546
2547	#D-#==================================================
2548	#D-function IGCM_sys_CountJobInQueue
2549	#D-* Purpose: Check if job_name is currently
2550	#D- running or in queue
2551	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
2552	#D-
2553	function IGCM_sys_CountJobInQueue {
2554	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
2555	if ( $DEBUG_sys ) ; then
2556	echo "IGCM_sys_CountJobInQueue"
2557	fi
2558
2559	#NbRun=$( ccc_mstat -f \| grep -c ${JobName} )
2560
2561	# With -f option, the full job name is given in the last column
2562	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
2563
2564	eval ${2}=${NbRun}
2565
2566	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
2567	}
2568
2569	#D-#==================================================
2570	#D-function IGCM_sys_ListJobInQueue
2571	#D-* Purpose: Check if job_name is currently
2572	#D- running or in queue
2573	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
2574	#D-
2575	function IGCM_sys_ListJobInQueue {
2576	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
2577	if ( $DEBUG_sys ) ; then
2578	echo "IGCM_sys_ListJobInQueue"
2579	fi
2580
2581	# With -f option, the full job name is given in the last column
2582	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
2583	'( $2 == User && \
2584	$NF != /TS/ && \
2585	$NF !~ /PACK/ && \
2586	$NF !~ /REBUILD/ && \
2587	$NF !~ /pack/ ) \
2588	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
2589
2590	eval set -A ${2} ${JobList[*]}
2591
2592	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
2593	}
2594
2595	##############################################################
2596	# NCO OPERATOR
2597
2598	#D-#==================================================
2599	#D-function IGCM_sys_ncap2
2600	#D-* Purpose: encapsulate ncap2 call so as to manage error code and retry
2601	#D-* Examples:
2602	#D-
2603	function IGCM_sys_ncap2 {
2604	IGCM_debug_PushStack "IGCM_sys_ncap2" $@
2605	if ( $DEBUG_sys ) ; then
2606	echo "IGCM_sys_ncap2 :" $@
2607	fi
2608
2609	typeset NB_ESSAI DELAI status i
2610	# number of tentative
2611	NB_ESSAI=3
2612	# time delay between tentative
2613	DELAI=2
2614
2615	i=0
2616	while [ $i -lt $NB_ESSAI ] ; do
2617	ncap2 -C "$@" > /tmp/out_command.$$ 2>&1
2618	status=$?
2619	if [ ${status} -gt 0 ] ; then
2620	IGCM_debug_Print 2 "IGCM_sys_ncap2 : error code ${status}"
2621	cat /tmp/out_command.$$
2622	\rm /tmp/out_command.$$
2623	IGCM_debug_Print 2 "IGCM_sys_ncap2 : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2624	sleep $DELAI
2625	else
2626	\rm /tmp/out_command.$$
2627	break
2628	fi
2629	(( i = i + 1 ))
2630	done
2631
2632	if [ ${status} -gt 0 ] ; then
2633	echo "IGCM_sys_ncap2 : ncap2 error"
2634	IGCM_debug_Exit "ncap2"
2635	fi
2636
2637	IGCM_debug_PopStack "IGCM_sys_ncap2"
2638	}
2639
2640	#D-#==================================================
2641	#D-function IGCM_sys_ncatted
2642	#D-* Purpose: encapsulate ncatted call so as to manage error code and retry
2643	#D-* Examples:
2644	#D-
2645	function IGCM_sys_ncatted {
2646	IGCM_debug_PushStack "IGCM_sys_ncatted" $@
2647	if ( $DEBUG_sys ) ; then
2648	echo "IGCM_sys_ncatted :" $@
2649	fi
2650
2651	typeset NB_ESSAI DELAI status i
2652	# number of tentative
2653	NB_ESSAI=3
2654	# time delay between tentative
2655	DELAI=2
2656
2657	i=0
2658	while [ $i -lt $NB_ESSAI ] ; do
2659	ncatted "$@" > /tmp/out_command.$$ 2>&1
2660	status=$?
2661	if [ ${status} -gt 0 ] ; then
2662	IGCM_debug_Print 2 "IGCM_sys_ncatted : error code ${status}"
2663	cat /tmp/out_command.$$
2664	\rm /tmp/out_command.$$
2665	IGCM_debug_Print 2 "IGCM_sys_ncatted : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2666	sleep $DELAI
2667	else
2668	\rm /tmp/out_command.$$
2669	break
2670	fi
2671	(( i = i + 1 ))
2672	done
2673
2674	if [ ${status} -gt 0 ] ; then
2675	echo "IGCM_sys_ncatted : ncatted error"
2676	IGCM_debug_Exit "ncatted"
2677	fi
2678
2679	IGCM_debug_PopStack "IGCM_sys_ncatted"
2680	}
2681
2682	#D-#==================================================
2683	#D-function IGCM_sys_ncbo
2684	#D-* Purpose: encapsulate ncbo call so as to manage error code and retry
2685	#D-* Examples:
2686	#D-
2687	function IGCM_sys_ncbo {
2688	IGCM_debug_PushStack "IGCM_sys_ncbo" $@
2689	if ( $DEBUG_sys ) ; then
2690	echo "IGCM_sys_ncbo :" $@
2691	fi
2692
2693	typeset NB_ESSAI DELAI status i
2694	# number of tentative
2695	NB_ESSAI=3
2696	# time delay between tentative
2697	DELAI=2
2698
2699	i=0
2700	while [ $i -lt $NB_ESSAI ] ; do
2701	ncbo -C $@ > /tmp/out_command.$$ 2>&1
2702	status=$?
2703	if [ ${status} -gt 0 ] ; then
2704	IGCM_debug_Print 2 "IGCM_sys_ncbo : error code ${status}"
2705	cat /tmp/out_command.$$
2706	\rm /tmp/out_command.$$
2707	IGCM_debug_Print 2 "IGCM_sys_ncbo : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2708	sleep $DELAI
2709	else
2710	\rm /tmp/out_command.$$
2711	break
2712	fi
2713	(( i = i + 1 ))
2714	done
2715
2716	if [ ${status} -gt 0 ] ; then
2717	echo "IGCM_sys_ncbo : ncbo error"
2718	IGCM_debug_Exit "ncbo"
2719	fi
2720
2721	IGCM_debug_PopStack "IGCM_sys_ncbo"
2722	}
2723
2724	#D-#==================================================
2725	#D-function IGCM_sys_ncdif
2726	#D-* Purpose: encapsulate ncdiff call so as to manage error code and retry
2727	#D-* Examples:
2728	#D-
2729	function IGCM_sys_ncdiff {
2730	IGCM_debug_PushStack "IGCM_sys_ncdiff" $@
2731	if ( $DEBUG_sys ) ; then
2732	echo "IGCM_sys_ncdiff :" $@
2733	fi
2734
2735	typeset NB_ESSAI DELAI status i
2736	# number of tentative
2737	NB_ESSAI=3
2738	# time delay between tentative
2739	DELAI=2
2740
2741	i=0
2742	while [ $i -lt $NB_ESSAI ] ; do
2743	ncdiff -C $@ > /tmp/out_command.$$ 2>&1
2744	status=$?
2745	if [ ${status} -gt 0 ] ; then
2746	IGCM_debug_Print 2 "IGCM_sys_ncdiff : error code ${status}"
2747	cat /tmp/out_command.$$
2748	\rm /tmp/out_command.$$
2749	IGCM_debug_Print 2 "IGCM_sys_ncdiff : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2750	sleep $DELAI
2751	else
2752	\rm /tmp/out_command.$$
2753	break
2754	fi
2755	(( i = i + 1 ))
2756	done
2757
2758	if [ ${status} -gt 0 ] ; then
2759	echo "IGCM_sys_ncdiff : ncdiff error"
2760	IGCM_debug_Exit "ncdiff"
2761	fi
2762
2763	IGCM_debug_PopStack "IGCM_sys_ncdiff"
2764	}
2765
2766	#D-#==================================================
2767	#D-function IGCM_sys_ncea
2768	#D-* Purpose: encapsulate ncea call so as to manage error code and retry
2769	#D-* Examples:
2770	#D-
2771	function IGCM_sys_ncea {
2772	IGCM_debug_PushStack "IGCM_sys_ncea" $@
2773	if ( $DEBUG_sys ) ; then
2774	echo "IGCM_sys_ncea :" $@
2775	fi
2776
2777	typeset NB_ESSAI DELAI status i
2778	# number of tentative
2779	NB_ESSAI=3
2780	# time delay between tentative
2781	DELAI=2
2782
2783	i=0
2784	while [ $i -lt $NB_ESSAI ] ; do
2785	ncea -C $@ > /tmp/out_command.$$ 2>&1
2786	status=$?
2787	if [ ${status} -gt 0 ] ; then
2788	IGCM_debug_Print 2 "IGCM_sys_ncea : error code ${status}"
2789	cat /tmp/out_command.$$
2790	\rm /tmp/out_command.$$
2791	IGCM_debug_Print 2 "IGCM_sys_ncea : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2792	sleep $DELAI
2793	else
2794	\rm /tmp/out_command.$$
2795	break
2796	fi
2797	(( i = i + 1 ))
2798	done
2799
2800	if [ ${status} -gt 0 ] ; then
2801	echo "IGCM_sys_ncea : ncea error"
2802	IGCM_debug_Exit "ncea"
2803	fi
2804
2805	IGCM_debug_PopStack "IGCM_sys_ncea"
2806	}
2807
2808	#D-#==================================================
2809	#D-function IGCM_sys_ncecat
2810	#D-* Purpose: encapsulate ncecat call so as to manage error code and retry
2811	#D-* Examples:
2812	#D-
2813	function IGCM_sys_ncecat {
2814	IGCM_debug_PushStack "IGCM_sys_ncecat" $@
2815	if ( $DEBUG_sys ) ; then
2816	echo "IGCM_sys_ncecat :" $@
2817	fi
2818
2819	typeset NB_ESSAI DELAI status i
2820	# number of tentative
2821	NB_ESSAI=3
2822	# time delay between tentative
2823	DELAI=2
2824
2825	i=0
2826	while [ $i -lt $NB_ESSAI ] ; do
2827	ncecat -C $@ > /tmp/out_command.$$ 2>&1
2828	status=$?
2829	if [ ${status} -gt 0 ] ; then
2830	IGCM_debug_Print 2 "IGCM_sys_ncecat : error code ${status}"
2831	cat /tmp/out_command.$$
2832	\rm /tmp/out_command.$$
2833	IGCM_debug_Print 2 "IGCM_sys_ncecat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2834	sleep $DELAI
2835	else
2836	\rm /tmp/out_command.$$
2837	break
2838	fi
2839	(( i = i + 1 ))
2840	done
2841
2842	if [ ${status} -gt 0 ] ; then
2843	echo "IGCM_sys_ncecat : ncecat error"
2844	IGCM_debug_Exit "ncecat"
2845	fi
2846
2847	IGCM_debug_PopStack "IGCM_sys_ncecat"
2848	}
2849
2850	#D-#==================================================
2851	#D-function IGCM_sys_ncflint
2852	#D-* Purpose: encapsulate ncflint call so as to manage error code and retry
2853	#D-* Examples:
2854	#D-
2855	function IGCM_sys_ncflint {
2856	IGCM_debug_PushStack "IGCM_sys_ncflint" $@
2857	if ( $DEBUG_sys ) ; then
2858	echo "IGCM_sys_ncflint :" $@
2859	fi
2860
2861	typeset NB_ESSAI DELAI status i
2862	# number of tentative
2863	NB_ESSAI=3
2864	# time delay between tentative
2865	DELAI=2
2866
2867	i=0
2868	while [ $i -lt $NB_ESSAI ] ; do
2869	ncflint -C $@ > /tmp/out_command.$$ 2>&1
2870	status=$?
2871	if [ ${status} -gt 0 ] ; then
2872	IGCM_debug_Print 2 "IGCM_sys_ncflint : error code ${status}"
2873	cat /tmp/out_command.$$
2874	\rm /tmp/out_command.$$
2875	IGCM_debug_Print 2 "IGCM_sys_ncflint : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2876	sleep $DELAI
2877	else
2878	\rm /tmp/out_command.$$
2879	break
2880	fi
2881	(( i = i + 1 ))
2882	done
2883
2884	if [ ${status} -gt 0 ] ; then
2885	echo "IGCM_sys_ncflint : ncflint error"
2886	IGCM_debug_Exit "ncflint"
2887	fi
2888
2889	IGCM_debug_PopStack "IGCM_sys_ncflint"
2890	}
2891
2892	#D-#==================================================
2893	#D-function IGCM_sys_ncks
2894	#D-* Purpose: encapsulate ncks call so as to manage error code and retry
2895	#D-* Examples:
2896	#D-
2897	function IGCM_sys_ncks {
2898	IGCM_debug_PushStack "IGCM_sys_ncks" $@
2899	if ( $DEBUG_sys ) ; then
2900	echo "IGCM_sys_ncks :" $@
2901	fi
2902
2903	typeset NB_ESSAI DELAI status i
2904	# number of tentative
2905	NB_ESSAI=3
2906	# time delay between tentative
2907	DELAI=2
2908
2909	i=0
2910	while [ $i -lt $NB_ESSAI ] ; do
2911	ncks -C $@ > /tmp/out_command.$$ 2>&1
2912	status=$?
2913	if [ ${status} -gt 0 ] ; then
2914	IGCM_debug_Print 2 "IGCM_sys_ncks : error code ${status}"
2915	cat /tmp/out_command.$$
2916	\rm /tmp/out_command.$$
2917	IGCM_debug_Print 2 "IGCM_sys_ncks : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2918	sleep $DELAI
2919	else
2920	\rm /tmp/out_command.$$
2921	break
2922	fi
2923	(( i = i + 1 ))
2924	done
2925
2926	if [ ${status} -gt 0 ] ; then
2927	echo "IGCM_sys_ncks : ncks error"
2928	IGCM_debug_Exit "ncks"
2929	fi
2930
2931	IGCM_debug_PopStack "IGCM_sys_ncks"
2932	}
2933
2934	#D-#==================================================
2935	#D-function IGCM_sys_ncpdq
2936	#D-* Purpose: encapsulate ncpdq call so as to manage error code and retry
2937	#D-* Examples:
2938	#D-
2939	function IGCM_sys_ncpdq {
2940	IGCM_debug_PushStack "IGCM_sys_ncpdq" $@
2941	if ( $DEBUG_sys ) ; then
2942	echo "IGCM_sys_ncpdq :" $@
2943	fi
2944
2945	typeset NB_ESSAI DELAI status i
2946	# number of tentative
2947	NB_ESSAI=3
2948	# time delay between tentative
2949	DELAI=2
2950
2951	i=0
2952	while [ $i -lt $NB_ESSAI ] ; do
2953	ncpdq -C $@ > /tmp/out_command.$$ 2>&1
2954	status=$?
2955	if [ ${status} -gt 0 ] ; then
2956	IGCM_debug_Print 2 "IGCM_sys_ncpdq : error code ${status}"
2957	cat /tmp/out_command.$$
2958	\rm /tmp/out_command.$$
2959	IGCM_debug_Print 2 "IGCM_sys_ncpdq : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
2960	sleep $DELAI
2961	else
2962	\rm /tmp/out_command.$$
2963	break
2964	fi
2965	(( i = i + 1 ))
2966	done
2967
2968	if [ ${status} -gt 0 ] ; then
2969	echo "IGCM_sys_ncpdq : ncpdq error"
2970	IGCM_debug_Exit "ncpdq"
2971	fi
2972
2973	IGCM_debug_PopStack "IGCM_sys_ncpdq"
2974	}
2975
2976	#D-#==================================================
2977	#D-function IGCM_sys_ncra
2978	#D-* Purpose: encapsulate ncra call so as to manage error code and retry
2979	#D-* Examples:
2980	#D-
2981	function IGCM_sys_ncra {
2982	IGCM_debug_PushStack "IGCM_sys_ncra" $@
2983	if ( $DEBUG_sys ) ; then
2984	echo "IGCM_sys_ncra :" $@
2985	fi
2986
2987	typeset NB_ESSAI DELAI status i
2988	# number of tentative
2989	NB_ESSAI=3
2990	# time delay between tentative
2991	DELAI=2
2992
2993	i=0
2994	while [ $i -lt $NB_ESSAI ] ; do
2995	ncra -C $@ > /tmp/out_command.$$ 2>&1
2996	status=$?
2997	if [ ${status} -gt 0 ] ; then
2998	IGCM_debug_Print 2 "IGCM_sys_ncra : error code ${status}"
2999	cat /tmp/out_command.$$
3000	\rm /tmp/out_command.$$
3001	IGCM_debug_Print 2 "IGCM_sys_ncra : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3002	sleep $DELAI
3003	else
3004	\rm /tmp/out_command.$$
3005	break
3006	fi
3007	(( i = i + 1 ))
3008	done
3009
3010	if [ ${status} -gt 0 ] ; then
3011	echo "IGCM_sys_ncra : ncra error"
3012	IGCM_debug_Exit "ncra"
3013	fi
3014
3015	IGCM_debug_PopStack "IGCM_sys_ncra"
3016	}
3017
3018	#D-#==================================================
3019	#D-function IGCM_sys_ncrcat
3020	#D-* Purpose: encapsulate ncrcat call so as to manage error code and retry
3021	#D-* Examples:
3022	#D-
3023	function IGCM_sys_ncrcat {
3024	IGCM_debug_PushStack "IGCM_sys_ncrcat" $@
3025	if ( $DEBUG_sys ) ; then
3026	echo "IGCM_sys_ncrcat :" $@
3027	fi
3028
3029	typeset NB_ESSAI DELAI status i lastArg
3030	# number of tentative
3031	NB_ESSAI=3
3032	# time delay between tentative
3033	DELAI=2
3034
3035	i=0
3036	while [ $i -lt $NB_ESSAI ] ; do
3037	ncrcat -C $@ > /tmp/out_command.$$ 2>&1
3038	status=$?
3039	if [ ${status} -gt 0 ] ; then
3040	IGCM_debug_Print 2 "IGCM_sys_ncrcat : error code ${status}"
3041	cat /tmp/out_command.$$
3042	\rm /tmp/out_command.$$
3043	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3044	sleep $DELAI
3045	elif [ ! "X$( grep "WARNING Intra-file non-monotonicity" /tmp/out_command.$$ )" = "X" ] ; then
3046	IGCM_debug_Print 2 "IGCM_sys_ncrcat : WARNING Intra-file non-monotonicity"
3047	cat /tmp/out_command.$$
3048	# remove files having corrupted time axis
3049	eval lastArg=\${$#}
3050	IGCM_debug_Print 2 "IGCM_sys_ncrcat : Delete ${lastArg}"
3051	\rm ${lastArg}
3052	\rm /tmp/out_command.$$
3053	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3054	sleep $DELAI
3055	else
3056	\rm /tmp/out_command.$$
3057	break
3058	fi
3059	(( i = i + 1 ))
3060	done
3061
3062	if [ ${status} -gt 0 ] ; then
3063	echo "IGCM_sys_ncrcat : ncrcat error"
3064	#IGCM_debug_Exit "ncrcat"
3065	fi
3066
3067	IGCM_debug_PopStack "IGCM_sys_ncrcat"
3068	}
3069
3070	#D-#==================================================
3071	#D-function IGCM_sys_ncrename
3072	#D-* Purpose: encapsulate ncrename call so as to manage error code and retry
3073	#D-* Examples:
3074	#D-
3075	function IGCM_sys_ncrename {
3076	IGCM_debug_PushStack "IGCM_sys_ncrename" $@
3077	if ( $DEBUG_sys ) ; then
3078	echo "IGCM_sys_ncrename :" $@
3079	fi
3080
3081	typeset NB_ESSAI DELAI status i
3082	# number of tentative
3083	NB_ESSAI=3
3084	# time delay between tentative
3085	DELAI=2
3086
3087	i=0
3088	while [ $i -lt $NB_ESSAI ] ; do
3089	ncrename $@ > /tmp/out_command.$$ 2>&1
3090	status=$?
3091	if [ ${status} -gt 0 ] ; then
3092	IGCM_debug_Print 2 "IGCM_sys_ncrename : error code ${status}"
3093	cat /tmp/out_command.$$
3094	\rm /tmp/out_command.$$
3095	IGCM_debug_Print 2 "IGCM_sys_ncrename : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3096	sleep $DELAI
3097	else
3098	\rm /tmp/out_command.$$
3099	break
3100	fi
3101	(( i = i + 1 ))
3102	done
3103
3104	if [ ${status} -gt 0 ] ; then
3105	echo "IGCM_sys_ncrename : ncrename error"
3106	IGCM_debug_Exit "ncrename"
3107	fi
3108
3109	IGCM_debug_PopStack "IGCM_sys_ncrename"
3110	}
3111
3112	#D-#==================================================
3113	#D-function IGCM_sys_ncwa
3114	#D-* Purpose: encapsulate ncwa call so as to manage error code and retry
3115	#D-* Examples:
3116	#D-
3117	function IGCM_sys_ncwa {
3118	IGCM_debug_PushStack "IGCM_sys_ncwa" $@
3119	if ( $DEBUG_sys ) ; then
3120	echo "IGCM_sys_ncwa :" $@
3121	fi
3122
3123	typeset NB_ESSAI DELAI status i
3124	# number of tentative
3125	NB_ESSAI=3
3126	# time delay between tentative
3127	DELAI=2
3128
3129	i=0
3130	while [ $i -lt $NB_ESSAI ] ; do
3131	ncwa -C $@ > /tmp/out_command.$$ 2>&1
3132	status=$?
3133	if [ ${status} -gt 0 ] ; then
3134	IGCM_debug_Print 2 "IGCM_sys_ncwa : error code ${status}"
3135	cat /tmp/out_command.$$
3136	\rm /tmp/out_command.$$
3137	IGCM_debug_Print 2 "IGCM_sys_ncwa : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
3138	sleep $DELAI
3139	else
3140	\rm /tmp/out_command.$$
3141	break
3142	fi
3143	(( i = i + 1 ))
3144	done
3145
3146	if [ ${status} -gt 0 ] ; then
3147	echo "IGCM_sys_ncwa : ncwa error"
3148	IGCM_debug_Exit "ncwa"
3149	fi
3150
3151	IGCM_debug_PopStack "IGCM_sys_ncwa"
3152	}
3153
3154	##############################################################
3155	# CDO OPERATOR
3156
3157	#D-#==================================================
3158	#D-function IGCM_sys_cdo
3159	#D-* Purpose: encapsulate cdo call so as to manage error code and retry
3160	#D-* Examples:
3161	#D-
3162	function IGCM_sys_cdo {
3163	IGCM_debug_PushStack "IGCM_sys_cdo" $@
3164	if ( $DEBUG_sys ) ; then
3165	echo "IGCM_sys_cdo :" $@
3166	fi
3167
3168	typeset status
3169
3170	\cdo $@ > /tmp/out_command.$$ 2>&1
3171	status=$?
3172	if [ ${status} -gt 0 ] ; then
3173	echo "IGCM_sys_cdo : error code ${status}"
3174	cat /tmp/out_command.$$
3175	\rm /tmp/out_command.$$
3176	IGCM_debug_PopStack "IGCM_sys_cdo"
3177	return 1
3178	else
3179	IGCM_debug_PopStack "IGCM_sys_cdo"
3180	return 0
3181	fi
3182
3183	IGCM_debug_PopStack "IGCM_sys_cdo"
3184	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: