Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_curie.ksh @ 1391

Last change on this file since 1391 was 1390, checked in by mafoipsl, 7 years ago
On curie : for MPMD mode add a parameter in config.card (ExecutionModeOnCurie?) to select ccc_mprun or mpirun method. mpirun is used by default. These methods are described here : wiki:IntegrationOpenMP/EnteteCommandesCurie?
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 60.6 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Curie
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=curie
71	# add default project on curie
72	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=gen0826} \| cut -d@ -f1 )
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=ccc_msub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Set environment tools (ferret, nco, cdo, rebuild, ...)
93	#====================================================
94	if [ X${TaskType} = Xcomputing ] ; then
95	module load python/2.7.8 > /dev/null 2>&1
96	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4.3.3.1_hdf5_parallel_curie_ksh > /dev/null 2>&1
97	# to run with netcdf 3.6.3 ie compilation done before 17/2/2014
98	# uncomment 2 lines :
99	# module unload netcdf
100	# module load netcdf/3.6.3
101	else
102	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4.3.3.1_hdf5_parallel_curie_ksh > /dev/null 2>&1
103	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
104	export UVCDAT_ANONYMOUS_LOG=FALSE
105	fi
106
107	# Use CMIP6 storage space when using CMIP6 cpu ressources
108	[[ "X${PROJECT}" = "Xdevcmip6" && ! $(module list --terse 2>&1 \| grep dfldatadir/gencmip6) ]] && module switch dfldatadir dfldatadir/gencmip6
109
110	# FYI
111	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
112	[ ! X${TaskType} = Xchecking ] && module list
113
114	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddNoise/src_X64_CURIE/bin
115	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddPerturbation/src_X64_CURIE/bin
116	export PATH=${PATH}:$( ccc_home -u p86ipsl )/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/
117
118	#====================================================
119	# Specific for ocean additionnal diagnostic
120	export FER_GO="$FER_GO /home/cont003/p86denv/IGCM_POST_UTIL/JNL /home/cont003/p86denv/GRAF /home/cont003/p86denv/GRAF/GO"
121	export FER_PALETTE="$FER_PALETTE /home/cont003/p86denv/GRAF/PALET"
122
123	#====================================================
124	# Host specific DIRECTORIES
125	#====================================================
126
127	# ============ CESIUM START ============ #
128
129	#====================================================
130	#- Mirror libIGCM from titane to cesium if needed
131	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
132	#if [ ! ${ROOTSYS} = "home" ] ; then
133	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
134	#else
135	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
136	#fi
137
138	#====================================================
139	#- libIGCM_POST
140	#if ( ${MirrorlibIGCM} ) ; then
141	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
142	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
143	#else
144	# typeset -r libIGCM_POST=${libIGCM}
145	#fi
146
147	# ============ CESIUM END ============ #
148
149	#====================================================
150	#- MirrorlibIGCM for frontend
151	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
152
153	#====================================================
154	#- libIGCM_POST for frontend
155	typeset -r libIGCM_POST=${libIGCM}
156
157	#====================================================
158	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
159	typeset -r R_EXE="${MODIPSL}/bin"
160
161	#====================================================
162	#- SUBMIT_DIR : submission dir
163	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
164
165	#====================================================
166	#- IN
167	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
168	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/ccc/work/cont003/dsm/p24data}
169
170	#====================================================
171	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
172	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
173
174	#====================================================
175	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
176	typeset -r OUTCOMMAND_PATH=/tmp
177
178	#====================================================
179	#- HOST_MPIRUN_COMMAND
180	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
181
182	#====================================================
183	#- Max number of arguments passed to nco operator or demigration command
184	UNIX_MAX_LIMIT=360
185
186	#====================================================
187	#- set PackDefault to true on curie
188	PackDefault=true
189
190	#====================================================
191	#- Default number of MPI task for IPSL coupled model
192	#- required for backward compatibility
193	#-
194	DEFAULT_NUM_PROC_OCE=5
195	DEFAULT_NUM_PROC_CPL=1
196	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
197
198	#D-#==================================================
199	#D-function IGCM_sys_defineArchives
200	#D-* Purpose:
201	#D-* Define ARCHIVE : Dedicated to large files
202	#D-* Define STORAGE : Dedicated to small/medium files
203	#D-* Define R_OUT : Output tree located on ARCHIVE
204	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
205	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
206	#D-* if SpaceName=TEST everything is stored on SCRATCHDIR
207	#D-* Examples:
208	#D-
209	function IGCM_sys_defineArchives {
210	IGCM_debug_PushStack "IGCM_sys_defineArchives"
211
212	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
213	#====================================================
214	#- ARCHIVE (dedicated to large files)
215	ARCHIVE=${config_UserChoices_ARCHIVE}
216	else
217	#====================================================
218	#- ARCHIVE (dedicated to large files)
219	ARCHIVE=${CCCSTOREDIR}
220	fi
221
222	if [ ! X${config_UserChoices_STORAGE} = X ]; then
223	#====================================================
224	#- STORAGE (dedicated to small/medium files)
225	STORAGE=${config_UserChoices_STORAGE}
226	else
227	#====================================================
228	#- STORAGE (dedicated to small/medium files)
229	STORAGE=${CCCWORKDIR}
230	fi
231
232	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
233	#====================================================
234	#- R_OUT
235	R_OUT=${SCRATCHDIR}/IGCM_OUT
236
237	#====================================================
238	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
239	R_FIG=${SCRATCHDIR}/IGCM_OUT
240
241	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
242
243	else
244	#====================================================
245	#- R_OUT
246	R_OUT=${ARCHIVE}/IGCM_OUT
247
248	#====================================================
249	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
250	R_FIG=${STORAGE}/IGCM_OUT
251	fi
252
253	#====================================================
254	#- R_BUF (ONLY FOR double copy an scratch)
255	R_BUF=${SCRATCHDIR}/IGCM_OUT
256
257	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
258	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
259	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
260
261	IGCM_debug_PopStack "IGCM_sys_defineArchives"
262	}
263
264	#D-#==================================================
265	#D-function IGCM_sys_RshArchive
266	#D-* Purpose: Archive rsh command
267	#D-* Examples:
268	#D-
269	function IGCM_sys_RshArchive {
270	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
271	/bin/ksh <<-EOF
272	${@}
273	EOF
274	status=$?
275	if [ ${status} -gt 0 ] ; then
276	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
277	IGCM_debug_Exit "IGCM_sys_RshArchive"
278	fi
279	IGCM_debug_PopStack "IGCM_sys_RshArchive"
280	}
281
282	#D-#==================================================
283	#D-function IGCM_sys_RshArchive_NoError
284	#D-* Purpose: Archive rsh command, without error
285	#D-* used only in monitoring.job
286	#D-* Examples:
287	#D-
288	function IGCM_sys_RshArchive_NoError {
289	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
290	/bin/ksh <<-EOF
291	${@} 2> /dev/null
292	EOF
293	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
294	}
295
296	#D-#==================================================
297	#D-function IGCM_sys_MkdirArchive
298	#D-* Purpose: Mkdir on Archive
299	#D-* Examples:
300	#D-
301	function IGCM_sys_MkdirArchive {
302	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
303	if ( $DEBUG_sys ) ; then
304	echo "IGCM_sys_MkdirArchive :" $@
305	fi
306	#- creation de repertoire sur le serveur fichier
307	if [ ! -d ${1} ]; then
308	\mkdir -p $1
309	status=$?
310
311	if [ ${status} -gt 0 ] ; then
312	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
313	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
314	fi
315	fi
316	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
317	}
318
319	#D-#==================================================
320	#D-function IGCM_sys_TestDirArchive
321	#D-* Purpose: Test Directory that must exists on Archive
322	#D-* Examples:
323	#D-
324	function IGCM_sys_TestDirArchive {
325	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
326	if ( $DEBUG_sys ) ; then
327	echo "IGCM_sys_TestDirArchive :" $@
328	fi
329	typeset ExistFlag
330	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
331	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
332	return ${ExistFlag}
333	}
334
335	#D-#==================================================
336	#D-function IGCM_sys_IsFileArchived
337	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
338	#D-* Examples:
339	#D-
340	function IGCM_sys_IsFileArchived {
341	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
342	if ( $DEBUG_sys ) ; then
343	echo "IGCM_sys_IsFileArchived :" $@
344	fi
345	typeset IsArchivedFlag
346	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
347	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
348
349	return ${IsArchivedFlag}
350	}
351
352	#D-#==================================================
353	#D-function IGCM_sys_TestFileArchive
354	#D-* Purpose: Test file that must NOT EXISTS on Archive
355	#D-* Examples:
356	#D-
357	function IGCM_sys_TestFileArchive {
358	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
359	typeset ExistFlag
360	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
361	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
362
363	return ${ExistFlag}
364	}
365
366	#D-#==================================================
367	#D-function IGCM_sys_CountFileArchive
368	#D-* Purpose: Count files on Archive filesystem
369	#D-* Examples:
370	#D-
371	function IGCM_sys_CountFileArchive {
372	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
373	ls ${@} 2>/dev/null \| wc -l
374	if [ $? -gt 0 ] ; then
375	echo "IGCM_sys_CountFileArchive : erreur."
376	fi
377	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
378	}
379
380	#D-#==================================================
381	#D-function IGCM_sys_Tree
382	#D-* Purpose: Tree directories with files on ${ARCHIVE}
383	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
384	#D-
385	function IGCM_sys_Tree {
386	IGCM_debug_PushStack "IGCM_sys_Tree" $@
387	if ( $DEBUG_sys ) ; then
388	echo "IGCM_sys_Tree :" $@
389	fi
390
391	\ls -lR ${@}
392
393	IGCM_debug_PopStack "IGCM_sys_Tree"
394	}
395
396	#D-#==================================================
397	#D-function IGCM_sys_Qsub
398	#D-* Purpose: Qsub new job
399	#D-* Examples:
400	#D-
401	function IGCM_sys_Qsub {
402	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
403	if ( $DEBUG_sys ) ; then
404	echo "IGCM_sys_Qsub :" $@
405	fi
406	typeset options status
407	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
408
409	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
410	status=$?
411
412	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
413	if [ ${status} -gt 0 ] ; then
414	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
415	IGCM_debug_Exit "IGCM_sys_Qsub"
416	else
417	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
418	fi
419	IGCM_debug_PopStack "IGCM_sys_Qsub"
420	}
421
422	#D-#==================================================
423	#D-function IGCM_sys_QsubPost
424	#D-* Purpose: Qsub new job on scalaire
425	#D-* Examples:
426	#D-
427	function IGCM_sys_QsubPost {
428	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
429	if ( $DEBUG_sys ) ; then
430	echo "IGCM_sys_QsubPost :" $@
431	fi
432	typeset options status
433
434	# EASIER TO DO THIS INSTEAD OF DUPLICATING libIGCM_sys_curie.ksh
435	case $( hostname -s ) in
436	curie*)
437	options="-Q normal -A ${PROJECT} -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out";;
438	airain*)
439	options="-q ivybridge -A dsm -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
440	esac
441
442	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
443	status=$?
444
445	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
446	if [ ${status} -gt 0 ] ; then
447	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
448	IGCM_debug_Exit "IGCM_sys_QsubPost"
449	else
450	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
451	fi
452	IGCM_debug_PopStack "IGCM_sys_QsubPost"
453	}
454
455	#D-*************************
456	#D- File transfer functions
457	#D-*************************
458	#D-
459
460	#D-#==================================================
461	#D-function IGCM_sys_RmRunDir
462	#D-* Purpose: rm tmpdir (dummy function most of the time batch
463	#D- scheduler will do the job)
464	#D-* Examples:
465	#D-
466	function IGCM_sys_RmRunDir {
467	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
468	if ( $DEBUG_sys ) ; then
469	echo "IGCM_sys_RmRunDir :" $@
470	fi
471
472	typeset status
473
474	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
475	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
476	status=$?
477
478	if [ ${status} -gt 0 ] ; then
479	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
480	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
481	IGCM_debug_Exit "IGCM_sys_RmRunDir"
482	else
483	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
484	fi
485	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
486	}
487
488	#D-#==================================================
489	#D-function IGCM_sys_Put_Dir
490	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
491	#D-* Examples:
492	#D-
493	function IGCM_sys_Put_Dir {
494	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
495	if ( $DEBUG_sys ) ; then
496	echo "IGCM_sys_Put_Dir :" $@
497	fi
498	if [ $DRYRUN = 0 ]; then
499	if [ ! -d ${1} ] ; then
500	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
501	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
502	return
503	fi
504
505	typeset status
506
507	# Only if we use rsync
508	#IGCM_sys_TestDirArchive $( dirname $2 )
509	#
510	#USUAL WAY
511	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
512	status=$?
513
514	if [ ${status} -gt 0 ] ; then
515	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
516	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
517	IGCM_debug_Exit "IGCM_sys_Put_Dir"
518	else
519	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
520	fi
521	fi
522	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
523	}
524
525	#D-#==================================================
526	#D-function IGCM_sys_Get_Dir
527	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
528	#D-* Examples:
529	#D-
530	function IGCM_sys_Get_Dir {
531	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
532	if ( $DEBUG_sys ) ; then
533	echo "IGCM_sys_Get_Dir :" $@
534	fi
535	if [ $DRYRUN = 0 ]; then
536	typeset NB_ESSAI DELAI status i
537	# number of tentative
538	NB_ESSAI=3
539	# time delay between tentative
540	DELAI=2
541
542	#
543	# USUAL WAY
544	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
545	ccc_hsm get -r $1
546
547	i=0
548	while [ $i -lt $NB_ESSAI ] ; do
549	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
550	status=$?
551	if [ ${status} -gt 0 ] ; then
552	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
553	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
554	sleep $DELAI
555	else
556	break
557	fi
558	(( i = i + 1 ))
559	done
560
561	if [ ${status} -gt 0 ] ; then
562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
563	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
564	IGCM_debug_Exit "IGCM_sys_Get_Dir"
565	else
566	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
567	fi
568	fi
569	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
570	}
571
572	#D-#==================================================
573	#D-function IGCM_sys_Put_Rest
574	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
575	#D- File and target directory must exist.
576	#D-* Examples:
577	#D-
578	function IGCM_sys_Put_Rest {
579	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
580	if ( $DEBUG_sys ) ; then
581	echo "IGCM_sys_Put_Rest :" $@
582	fi
583	if [ $DRYRUN = 0 ]; then
584	if [ ! -f ${1} ] ; then
585	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
586	IGCM_debug_Exit "IGCM_sys_Put_Rest"
587	fi
588
589	typeset status
590	#
591	# USUAL WAY
592	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593	status=$?
594
595	# #RSYNC WITH NETWORK SSH CALL
596	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
597	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
598
599	# #RSYNC WITH NFS USE
600	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
601	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
602
603	# status=$?
604	# IGCM_sys_Rsync_out $status
605
606	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
607	# (( status=status+$? ))
608
609	if [ ${status} -gt 0 ] ; then
610	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
611	[ -f ${1} ] && ls -l ${1}
612	[ -f ${2} ] && ls -l ${2}
613	[ -f ${2}/${1} ] && ls -l ${2}/${1}
614	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
615	IGCM_debug_Exit "IGCM_sys_Put_Rest"
616	else
617
618	if [ X${JobType} = XRUN ] ; then
619	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
620	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
621	fi
622
623	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
624	fi
625	fi
626	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
627	}
628
629	#D-#==================================================
630	#D-function IGCM_sys_Put_Out
631	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
632	#D-* Examples:
633	#D-
634	function IGCM_sys_Put_Out {
635	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
636	if ( $DEBUG_sys ) ; then
637	echo "IGCM_sys_Put_Out :" $@
638	fi
639
640	typeset NB_ESSAI DELAI status i exist skip
641	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
642
643	# number of tentative
644	NB_ESSAI=3
645	# time delay between tentative
646	DELAI=2
647
648	if [ $DRYRUN = 0 ]; then
649	if [ ! -f ${1} ] ; then
650	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
651	IGCM_debug_PopStack "IGCM_sys_Put_Out"
652	return 1
653	fi
654	#
655	IGCM_sys_MkdirArchive $( dirname $2 )
656	#
657	exist=false
658	skip=false
659	if [ -f $2 ] ; then
660	IGCM_debug_Print 1 "$2 already exist"
661	ccc_hsm get $2
662	exist=true
663	if [ "X$( diff $1 $2 )" = X ] ; then
664	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
665	skip=true
666	else
667	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
668	skip=false
669	fi
670	fi
671	#
672	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
673	IGCM_sys_Chmod u+w $2
674	fi
675
676	if [ X${skip} = Xfalse ] ; then
677	i=0
678	while [ $i -lt $NB_ESSAI ] ; do
679	# Identify file system
680	fileDeviceNumberInHex=$( stat -c %d $1 )
681	status=$?
682	if [ ${status} -gt 0 ] ; then
683	IGCM_debug_Exit "IGCM_sys_Put_Out"
684	fi
685	# Identify file system
686	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
687	status=$?
688	if [ ${status} -gt 0 ] ; then
689	IGCM_debug_Exit "IGCM_sys_Put_Out"
690	fi
691
692	if [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] ; then
693	# They are not on the same device. USUAL WAY
694	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
695	status=$?
696	else
697	# They are on the same device. NOT SO USUAL WAY
698	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
699	status=$?
700	fi
701	if [ ${status} -gt 0 ]; then
702	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
703	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
704	[ -f ${1} ] && ls -l ${1}
705	[ -f ${2} ] && ls -l ${2}
706	[ -f ${2}/${1} ] && ls -l ${2}/${1}
707	sleep $DELAI
708	else
709	break
710	fi
711	(( i = i + 1 ))
712	done
713	fi
714
715	# #RSYNC WITH NETWORK SSH CALL
716	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
717	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
718
719	# #RSYNC WITH NFS USE
720	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
721	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
722
723	# status=$?
724	# IGCM_sys_Rsync_out $status
725
726	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
727	# (( status=status+$? ))
728
729	if [ ${status} -gt 0 ] ; then
730	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
731	[ -f ${1} ] && ls -l ${1}
732	[ -f ${2} ] && ls -l ${2}
733	[ -f ${2}/${1} ] && ls -l ${2}/${1}
734	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
735	IGCM_debug_Exit "IGCM_sys_Put_Out"
736	else
737
738	if [ X${JobType} = XRUN ] ; then
739	if [ X${3} = X ] ; then
740	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
741	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
742	fi
743	fi
744
745	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
746	fi
747	fi
748	IGCM_debug_PopStack "IGCM_sys_Put_Out"
749	return 0
750	}
751
752	#D-#==================================================
753	#D-function IGCM_sys_Get
754	#D-* Purpose: Get a file from ${ARCHIVE}
755	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
756	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
757	function IGCM_sys_Get {
758	IGCM_debug_PushStack "IGCM_sys_Get" $@
759
760	typeset DEST dm_liste target file_work
761	typeset NB_ESSAI DELAI status i
762
763	if ( $DEBUG_sys ) ; then
764	echo "IGCM_sys_Get :" $@
765	fi
766
767	# number of tentative
768	NB_ESSAI=3
769	# time delay between tentative
770	DELAI=2
771
772	if [ $DRYRUN -le 2 ]; then
773	if [ X${1} = X'/l' ] ; then
774	eval set +A dm_liste \${${2}}
775	else
776	eval set +A dm_liste ${1}
777	fi
778	eval DEST=\${${#}}
779	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
780	status=$?
781	if [ ${status} -gt 0 ] ; then
782	echo "WARNING IGCM_sys_Get : error code ${status}"
783	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
784	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
785	fi
786
787	#if [ ${status} -gt 0 ] ; then
788	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
789	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
790	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
791	# sleep 30
792	# echo "We try another time"
793	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
794	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
795	# status=$?
796	# if [ ${status} -gt 0 ] ; then
797	# echo "ERROR IGCM_sys_Get : again demigration error :"
798	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
799	# IGCM_debug_Exit "IGCM_sys_Get"
800	# fi
801	# else
802	# echo "ERROR IGCM_sys_Get : demigration error :"
803	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
804	# IGCM_debug_Exit "IGCM_sys_Get"
805	# fi
806	#fi
807
808	# #RSYNC WITH NETWORK SSH CALL
809	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
810	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
811
812	# #RSYNC WITH NFS USE
813	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
814	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
815
816	# status=$?
817	# IGCM_sys_Rsync_out $status
818
819	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
820	# (( status=status+$? ))
821
822	#USUAL WAY
823	if [ X${1} = X'/l' ] ; then
824	for target in ${dm_liste[*]} ; do
825	local_file=$( basename ${target} )
826	# test if the target file is present before the loop
827	IGCM_sys_TestFileArchive ${target}
828	status=$?
829	if [ ${status} -gt 0 ] ; then
830	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
831	IGCM_debug_Exit "IGCM_sys_Get"
832	else
833	i=0
834	while [ $i -lt $NB_ESSAI ] ; do
835	#if [ X${DoLink} = Xtrue ] ; then
836	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
837	# status=$?
838	# else
839	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
840	# status=$?
841	#fi
842	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
843	status=$?
844	if [ ${status} -gt 0 ]; then
845	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
846	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
847	sleep $DELAI
848	else
849	break
850	fi
851	(( i = i + 1 ))
852	done
853	if [ ${status} -gt 0 ] ; then
854	echo "IGCM_sys_Get : error"
855	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
856	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
857	IGCM_debug_Exit "IGCM_sys_Get"
858	else
859	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
860	fi
861	fi
862	done
863	else
864	i=0
865	while [ $i -lt $NB_ESSAI ] ; do
866	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
867	status=$?
868	if [ ${status} -gt 0 ]; then
869	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
870	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
871	sleep $DELAI
872	else
873	break
874	fi
875	(( i = i + 1 ))
876	done
877	if [ ${status} -gt 0 ] ; then
878	echo "IGCM_sys_Get : error"
879	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
880	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
881	IGCM_debug_Exit "IGCM_sys_Get"
882	else
883	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
884	fi
885	fi
886	fi
887	IGCM_debug_PopStack "IGCM_sys_Get"
888	}
889
890	#D-#==================================================
891	#D-function IGCM_sys_GetDate_Monitoring
892	#D-* Purpose: get the last year for which the monitoring has been computed
893	#D-* Examples:
894	#D-
895	function IGCM_sys_GetDate_Monitoring {
896	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
897	if ( $DEBUG_sys ) ; then
898	echo "IGCM_sys_GetDate_Monitoring :" $@
899	fi
900
901	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
902
903	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
904	}
905
906	#D-#==================================================
907	#D-function IGCM_sys_Dods_Rm
908	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
909	#D-* Examples:
910	#D-
911	function IGCM_sys_Dods_Rm {
912	if ( $DEBUG_sys ) ; then
913	echo "IGCM_sys_Dods_Rm :" $@
914	fi
915	typeset status
916	if [ $DRYRUN = 0 ]; then
917
918	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
919	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
920	# echo "Nothing has been done."
921	# return
922	# fi
923
924	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
925	status=$?
926
927	# if [ ${status} -gt 0 ] ; then
928	# echo "IGCM_sys_Dods_Rm : error."
929	# cat out_dods_rm
930	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
931	# else
932	# rm out_dods_rm
933	# fi
934
935	fi
936	return $status
937	}
938
939	#D-#==================================================
940	#D-function IGCM_sys_Dods_Cp
941	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
942	#D-* Examples:
943	#D-
944	function IGCM_sys_Dods_Cp {
945	if ( $DEBUG_sys ) ; then
946	echo "IGCM_sys_Dods_Cp :" $@
947	fi
948	typeset status
949	if [ $DRYRUN = 0 ]; then
950
951	# if [ ! -d ${R_SAVE}/${1} ] ; then
952	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
953	# echo "Nothing has been done."
954	# return
955	# fi
956
957	/ccc/cont003/home/dsm/p86ipsl/bin/dods_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
958	status=$?
959
960	# if [ ${status} -gt 0 ] ; then
961	# echo "IGCM_sys_Dods_Cp : error."
962	# cat out_dods_cp
963	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
964	# else
965	# rm out_dods_cp
966	# fi
967
968	fi
969	return $status
970	}
971
972	#D-#==================================================
973	#D-function IGCM_sys_Put_Dods
974	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
975	#D-* Examples:
976	#D-
977	function IGCM_sys_Put_Dods {
978	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
979	if ( $DEBUG_sys ) ; then
980	echo "IGCM_sys_Put_Dods :" $@
981	fi
982	typeset status
983	if [ $DRYRUN = 0 ]; then
984	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
985	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
986	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
987	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
988	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
989	return
990	fi
991
992	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
993	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
994	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
995	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
996	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
997	return
998	fi
999	#
1000	if [ -d ${R_SAVE}/${1} ] ; then
1001	cd ${R_SAVE}
1002	elif [ -d ${R_FIGR}/${1} ] ; then
1003	cd ${R_FIGR}
1004	fi
1005
1006	IGCM_sys_Dods_Rm ${1}
1007	IGCM_sys_Dods_Cp ${1}
1008	status=0
1009
1010	if [ ${status} -gt 0 ] ; then
1011	echo "IGCM_sys_Put_Dods : error."
1012	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1013	fi
1014	fi
1015	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1016	}
1017
1018	##############################################################
1019	# REBUILD OPERATOR
1020
1021	#D-#==================================================
1022	#D-function IGCM_sys_sync
1023	#D-* Purpose: flush buffer on disk
1024	#D-* Examples:
1025	#D-
1026	function IGCM_sys_sync {
1027	IGCM_debug_PushStack "IGCM_sys_sync" $@
1028	if ( $DEBUG_sys ) ; then
1029	echo "IGCM_sys_sync :" $@
1030	fi
1031
1032	/bin/sync
1033
1034	IGCM_debug_PopStack "IGCM_sys_sync"
1035	}
1036
1037	############################################################
1038	# Activate Running Environnment Variables
1039
1040	#D-#==================================================
1041	#D-function IGCM_sys_activ_variables
1042	#D-* Purpose: set environement variables prior to execution
1043	#D-* Examples:
1044	#D-
1045	function IGCM_sys_activ_variables {
1046	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1047	if ( $DEBUG_sys ) ; then
1048	echo "IGCM_sys_activ_variables"
1049	fi
1050
1051	# --------------------------------------------------------------------
1052	#D- MPI specifications
1053	# --------------------------------------------------------------------
1054
1055	# --------------------------------------------------------------------
1056	#D- Other specifications
1057	# --------------------------------------------------------------------
1058
1059	ulimit -s unlimited
1060
1061	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1062	}
1063
1064	############################################################
1065	# Desactivate Running Environnment Variables
1066
1067	#D-#==================================================
1068	#D-function IGCM_sys_desactiv_variables
1069	#D-* Purpose: unset environement variables after execution
1070	#D-* Examples:
1071	#D-
1072	function IGCM_sys_desactiv_variables {
1073	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1074	if ( $DEBUG_sys ) ; then
1075	echo "IGCM_sys_desactiv_variables"
1076	fi
1077	# --------------------------------------------------------------------
1078	#D- MPI specifications
1079	# --------------------------------------------------------------------
1080
1081	# --------------------------------------------------------------------
1082	#D- Other specifications
1083	# --------------------------------------------------------------------
1084
1085	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1086	}
1087
1088	############################################################
1089	# Update job headers to be used by the scheduler
1090
1091	#D-#==================================================
1092	#D-function IGCM_sys_updateHeaders
1093	#D-* Purpose: Update job headers to be used by the scheduler
1094	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1095	#D-
1096	function IGCM_sys_updateHeaders {
1097	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1098	if ( $DEBUG_sys ) ; then
1099	echo "IGCM_sys_updateHeaders"
1100	fi
1101	typeset file
1102	file=$1
1103
1104	if [ ${executionType} -eq 1 ] ; then
1105	# MPMD + MPI
1106	sed -e "/::openMPthreads::/d" \
1107	-e "s/::JobNumProcTot::/${coreNumber}/" \
1108	-e "/#MSUB -x/d" \
1109	-e "/--cpu_bind=none/d" \
1110	${file} > ${file}.tmp
1111
1112	elif [ ${executionType} -eq 2 ] ; then
1113	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1114	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1115	sed -e "/::openMPthreads::/d" \
1116	-e "s/::JobNumProcTot::/${coreNumber}/" \
1117	${file} > ${file}.tmp
1118	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1119	sed -e "/::openMPthreads::/d" \
1120	-e "s/::JobNumProcTot::/${coreNumber}/" \
1121	-e "/--cpu_bind=none/d" \
1122	${file} > ${file}.tmp
1123	else
1124	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1125	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1126	fi
1127	elif [ ${executionType} -eq 3 ] ; then
1128	# SPMD + MPI/OMP
1129	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1130	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1131	-e "/#MSUB -x/d" \
1132	-e "/--cpu_bind=none/d" \
1133	${file} > ${file}.tmp
1134
1135	elif [ ${executionType} -eq 4 ] ; then
1136	# SPMD + MPI only
1137	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1138	-e "/::openMPthreads::/d" \
1139	-e "/#MSUB -x/d" \
1140	-e "/--cpu_bind=none/d" \
1141	${file} > ${file}.tmp
1142
1143	elif [ ${executionType} -eq 5 ] ; then
1144	# SPMD + OMP only
1145	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1146	-e "/::JobNumProcTot::/d" \
1147	-e "/#MSUB -x/d" \
1148	-e "/--cpu_bind=none/d" \
1149	${file} > ${file}.tmp
1150
1151	elif [ ${executionType} -eq 6 ] ; then
1152	# SEQUENTIAL THEN
1153	sed -e "s/::JobNumProcTot::/1/" \
1154	-e "/::openMPthreads::/d" \
1155	-e "/#MSUB -x/d" \
1156	-e "/--cpu_bind=none/d" \
1157	${file} > ${file}.tmp
1158
1159	fi
1160
1161	IGCM_sys_Mv ${file}.tmp ${file}
1162
1163	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1164	}
1165
1166	############################################################
1167	# Build MPI/OMP scripts run file (dummy function)
1168
1169	#D-#==================================================
1170	#D-function IGCM_sys_build_run_file
1171	#D-* Purpose: build run file (deprecated)
1172	#D-* Examples:
1173	#D-
1174	function IGCM_sys_build_run_file {
1175	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1176	}
1177
1178	############################################################
1179	# Build MPI/OMP scripts
1180
1181	#D-#==================================================
1182	#D-function IGCM_sys_build_execution_scripts
1183	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1184	#D-* Examples:
1185	#D-
1186	function IGCM_sys_build_execution_scripts
1187	{
1188	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1189	if ( $DEBUG_sys ) ; then
1190	echo "IGCM_sys_build_execution_scripts " $@
1191	fi
1192
1193	EXECUTION=${HOST_MPIRUN_COMMAND}
1194
1195	if ( ${OK_PARA_MPMD} ) ; then
1196
1197	# MPMD mode
1198	# 1 MPI only
1199	# 2 MPI/OpenMP mpirun method
1200	# 3 MPI/OpenMP ccc_mprun method
1201
1202	if [ -f run_file ] ; then
1203	IGCM_sys_Rm -f run_file
1204	fi
1205	touch run_file
1206
1207	# case 1 : Only MPI (MPMD)
1208	if ( ! ${OK_PARA_OMP} ) ; then
1209
1210	# Build run_file
1211
1212	# First loop on the components for the coupler ie oasis (only if oasis3)
1213	# the coupler ie oasis3 must be the first one
1214	for comp in ${config_ListOfComponents[*]} ; do
1215
1216	eval ExeNameIn=\${config_Executable_${comp}[0]}
1217	eval ExeNameOut=\${config_Executable_${comp}[1]}
1218
1219	# for CPL component only
1220	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1221
1222	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1223	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1224	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1225	fi
1226	done
1227
1228	# Then second loop on the components
1229	for comp in ${config_ListOfComponents[*]} ; do
1230
1231	eval ExeNameIn=\${config_Executable_${comp}[0]}
1232	eval ExeNameOut=\${config_Executable_${comp}[1]}
1233
1234	# Only if we really have an executable for the component and not the coupler ie oasis:
1235	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1236
1237	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1238	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1239	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1240	fi
1241	done
1242
1243	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1244
1245	IGCM_sys_Chmod u+x run_file
1246	if ( $DEBUG_sys ) ; then
1247	echo "run_file contains : "
1248	cat run_file
1249	fi
1250
1251	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1252
1253	# 2 MPI/OpenMP mpirun method
1254
1255	# Use of mpirun instead of ccc_mprun
1256	EXECUTION="time mpirun"
1257
1258	# Hosts treatment
1259	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1260
1261	i=0
1262	rm -f hosts rankfile
1263	IGCM_debug_Print 1 "sys Curie, Hosts available :"
1264	for nodes in `cat hosts.tmp`
1265	do
1266	host[$i]=$nodes
1267	echo "${host[$i]}" >> hosts
1268	IGCM_debug_Print 1 ${host[$i]}
1269	i=$((i+1))
1270	done
1271	rm -f hosts.tmp
1272
1273	listnodes=${host[*]}
1274
1275	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1276
1277	# Initialisation
1278	rank=0
1279	current_core=0
1280	core_per_node=16
1281	init_exec=n
1282
1283	# Loop on the components
1284	for comp in ${config_ListOfComponents[*]} ; do
1285
1286	eval ExeNameIn=\${config_Executable_${comp}[0]}
1287	eval ExeNameOut=\${config_Executable_${comp}[1]}
1288
1289	# Not possible if oasis has an executable (i.e old version of oasis3)
1290	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1291	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1292	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1293	IGCM_debug_Verif_Exit
1294	fi
1295
1296	# Only if we really have an executable for the component :
1297	if [ "X${ExeNameOut}" != X\"\" ] ; then
1298
1299	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1300	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1301
1302	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1303	echo "" >> script_${ExeNameOut}.ksh
1304	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1305
1306	# Check if the number of threads is correct
1307	case ${comp_proc_omp_loc} in
1308	2\|4\|8\|16)
1309	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1310	;;
1311	*)
1312	IGCM_debug_Exit "ERROR with OMP parameters !"
1313	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1314	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1315	IGCM_debug_Verif_Exit
1316	;;
1317	esac
1318	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1319	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1320	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1321	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1322	fi
1323
1324	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1325	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1326	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1327	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1328	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1329
1330	if [ ${init_exec} = y ] ; then
1331	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1332	else
1333	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1334	init_exec=y
1335	fi
1336
1337	# Build rankfile : method used to assign cores and nodes for the MPI process
1338	# Ex :
1339	#rank 0=curie5296 slot=0,1,2,3
1340	#rank 1=curie5296 slot=4,5,6,7
1341	# Example of final command :
1342	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1343	# with script_lmdz.x.ksh :
1344	# #!/bin/ksh
1345	#export KMP_STACKSIZE=3g
1346	#export KMP_LIBRARY=turnaround
1347	#export MKL_SERIAL=YES
1348	#OMP_NUM_THREADS=4
1349	#./lmdz.x
1350
1351	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1352	(( index_host = current_core / core_per_node ))
1353	host_value=${host[${index_host}]}
1354	(( slot = current_core % core_per_node ))
1355	virg=","
1356	string_final=""
1357	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1358	string=$index$virg
1359	string_final=$string_final$string
1360	done
1361	string_final=$( echo $string_final \| sed "s/.$//" )
1362	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1363	(( rank = rank + 1 ))
1364	(( current_core = current_core + comp_proc_omp_loc ))
1365	done
1366	fi
1367	done
1368
1369	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1370
1371	# 3 MPI/OpenMP ccc_mprun method
1372
1373	# MPI-OpenMP (MPMD)
1374	# example of run_file
1375	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1376	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1377	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1378
1379	# Loop on the components
1380	for comp in ${config_ListOfComponents[*]} ; do
1381
1382	eval ExeNameIn=\${config_Executable_${comp}[0]}
1383	eval ExeNameOut=\${config_Executable_${comp}[1]}
1384
1385	# Not possible if oasis has an executable (i.e old version of oasis3)
1386	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1387	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1388	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1389	IGCM_debug_Verif_Exit
1390	fi
1391
1392	# Only if we really have an executable for the component :
1393	if [ "X${ExeNameOut}" != X\"\" ] ; then
1394
1395	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1396	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1397
1398	# Build script files
1399
1400	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1401	echo "" >> script_${ExeNameOut}.ksh
1402	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1403
1404	# Check if the number of threads is correct
1405	case ${comp_proc_omp_loc} in
1406	2\|4\|8\|16)
1407	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1408	;;
1409	*)
1410	IGCM_debug_Exit "ERROR with OMP parameters !"
1411	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1412	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1413	IGCM_debug_Verif_Exit
1414	;;
1415	esac
1416	### only for LMDZ?
1417	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1418	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1419	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1420	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1421
1422	fi
1423
1424	# to have out/err per process on different files
1425	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1426	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1427
1428	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1429
1430	# Complete run_file
1431
1432	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1433
1434	fi
1435	done
1436
1437	## variable added to stop after 60s instead of 600s by default.
1438	## This is used when no error comes from executables and when something stopped an executable without notice.
1439	export SLURM_WAIT=60
1440
1441	## module advised by TGCC (instead of 2 variables)
1442	module load feature/bridge/heterogenous_mpmd
1443
1444	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1445
1446	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1447
1448	IGCM_sys_Chmod u+x run_file
1449	if ( $DEBUG_sys ) ; then
1450	echo "run_file contains : "
1451	cat run_file
1452	fi
1453
1454	else
1455
1456	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1457
1458	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1459
1460	else
1461	# Only one executable (SPMD mode).
1462
1463	for comp in ${config_ListOfComponents[*]} ; do
1464
1465	# Only if we really have an executable for the component :
1466	eval ExeNameOut=\${config_Executable_${comp}[1]}
1467	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1468
1469	# Build script files
1470
1471	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1472	echo "" >> script_${ExeNameOut}.ksh
1473	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1474
1475	if ( ${OK_PARA_OMP} ) ; then
1476	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1477	echo "" >> script_${ExeNameOut}.ksh
1478	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1479	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1480	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1481	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1482	fi
1483
1484	if ( ${OK_PARA_MPI} ) ; then
1485	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1486	# Default : ccc_mprun used if nb_proc gt 1
1487	# to have out/err per process on different files
1488	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1489	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1490	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1491	else
1492	# Default : ccc_mprun is NOT used if nb_proc eq 1
1493	# to have out/err per process on different files
1494	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1495	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1496	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1497	fi
1498
1499	IGCM_debug_Print 1 "sys Curie : script_${ExeNameOut}.ksh contains"
1500	cat script_${ExeNameOut}.ksh
1501
1502	fi
1503
1504	done
1505
1506	fi # ${OK_PARA_MPMD}
1507
1508	IGCM_debug_Print 1 "sys Curie : execution command is "
1509	IGCM_debug_Print 1 "$EXECUTION"
1510
1511	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1512	}
1513
1514	#D-#==================================================
1515	#D-function IGCM_sys_check_path
1516	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1517	#D-* do not point to an important use directory. Stop immediately in that case.
1518	#D-* Examples:
1519	#D-
1520	function IGCM_sys_check_path {
1521	IGCM_debug_PushStack "IGCM_sys_check_path"
1522	if ( $DEBUG_sys ) ; then
1523	echo "IGCM_sys_check_path"
1524	fi
1525
1526	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1527	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1528	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1529	IGCM_debug_Exit "This will stop the job"
1530	fi
1531	IGCM_debug_PopStack "IGCM_sys_check_path"
1532	}
1533
1534	#D-#==================================================
1535	#D-function IGCM_sys_check_quota
1536	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1537	#D-* Examples:
1538	#D-
1539	function IGCM_sys_check_quota {
1540	IGCM_debug_PushStack "IGCM_sys_check_quota"
1541	if ( $DEBUG_sys ) ; then
1542	echo "IGCM_sys_check_quota"
1543	fi
1544	# Limit of quota (in %)
1545	limit_quota=90
1546
1547	# Check of the volume
1548	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1549	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1550
1551	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1552
1553	unit_avail=${volume_avail: -1}
1554	unit_quota=${volume_quota: -1}
1555
1556	if [ "${unit_quota}" = "*" ] ; then
1557	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1558	IGCM_debug_Print 1 "More than 100% of your quota is used"
1559	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1560	IGCM_debug_Print 1 "You must have more than 10% available to run"
1561	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1562	IGCM_debug_Verif_Exit
1563	fi
1564
1565	temp_avail=${volume_avail%%${volume_avail: -1}*}
1566	temp_quota=${volume_quota%%${volume_quota: -1}*}
1567
1568	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1569
1570	# Convertion
1571	if [ ${volume_avail: -1} = "T" ] ; then
1572	(( temp_avail = temp_avail * 1000000000000 ))
1573	elif [ ${volume_avail: -1} = "G" ] ; then
1574	(( temp_avail = temp_avail * 1000000000 ))
1575	elif [ ${volume_avail: -1} = "M" ] ; then
1576	(( temp_avail = temp_avail * 1000000 ))
1577	elif [ ${volume_avail: -1} = "k" ] ; then
1578	(( temp_avail = temp_avail * 1000 ))
1579	else
1580	(( temp_avail = volume_avail ))
1581	fi
1582	if [ ${volume_quota: -1} = "T" ] ; then
1583	(( temp_quota = temp_quota * 1000000000000 ))
1584	elif [ ${volume_quota: -1} = "G" ] ; then
1585	(( temp_quota = temp_quota * 1000000000 ))
1586	elif [ ${volume_quota: -1} = "M" ] ; then
1587	(( temp_quota = temp_quota * 1000000 ))
1588	elif [ ${volume_quota: -1} = "k" ] ; then
1589	(( temp_quota = temp_quota * 1000 ))
1590	else
1591	(( temp_quota = volume_quota ))
1592	fi
1593	fi
1594
1595	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1596	# echo "volume ratio is " $quota_volume
1597
1598	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1599	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1600	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1601	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1602	IGCM_debug_Print 1 "You must have more than 10% available to run"
1603	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1604	IGCM_debug_Verif_Exit
1605	fi
1606
1607	fi
1608
1609	# Check of the number of inodes
1610
1611	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1612	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1613
1614	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1615
1616	unit_avail=${inode_avail: -1}
1617	unit_quota=${inode_quota: -1}
1618
1619	if [ "${unit_quota}" = "*" ] ; then
1620	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1621	IGCM_debug_Print 1 "More than 100% of your quota is used"
1622	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1623	IGCM_debug_Print 1 "You must have more than 10% available to run"
1624	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1625	IGCM_debug_Verif_Exit
1626	fi
1627
1628	temp_avail=${inode_avail%%${inode_avail: -1}*}
1629	temp_quota=${inode_quota%%${inode_quota: -1}*}
1630
1631	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1632
1633	# Convertion
1634	if [ ${inode_avail: -1} = "T" ] ; then
1635	(( temp_avail = temp_avail * 1000000000000 ))
1636	elif [ ${inode_avail: -1} = "G" ] ; then
1637	(( temp_avail = temp_avail * 1000000000 ))
1638	elif [ ${inode_avail: -1} = "M" ] ; then
1639	(( temp_avail = temp_avail * 1000000 ))
1640	elif [ ${inode_avail: -1} = "k" ] ; then
1641	(( temp_avail = temp_avail * 1000 ))
1642	else
1643	(( temp_avail = inode_avail ))
1644	fi
1645
1646	if [ ${inode_quota: -1} = "T" ] ; then
1647	(( temp_quota = temp_quota * 1000000000000 ))
1648	elif [ ${inode_quota: -1} = "G" ] ; then
1649	(( temp_quota = temp_quota * 1000000000 ))
1650	elif [ ${inode_quota: -1} = "M" ] ; then
1651	(( temp_quota = temp_quota * 1000000 ))
1652	elif [ ${inode_quota: -1} = "k" ] ; then
1653	(( temp_quota = temp_quota * 1000 ))
1654	else
1655	(( temp_quota = inode_quota ))
1656	fi
1657	fi
1658	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1659	# echo "inode ratio is " $quota_inode
1660
1661	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1662	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1663	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1664	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1665	IGCM_debug_Print 1 "You must have more than 10% available to run"
1666	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1667	IGCM_debug_Verif_Exit
1668	fi
1669	fi
1670	IGCM_debug_PopStack "IGCM_sys_check_quota"
1671	}
1672
1673	#D-#==================================================
1674	#D-function IGCM_sys_projectAccounting
1675	#D-* Purpose: store project accounting information in a file
1676	#D-* Examples:
1677	#D-
1678	function IGCM_sys_projectAccounting {
1679	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1680	if ( $DEBUG_sys ) ; then
1681	echo "IGCM_sys_check_quota"
1682	fi
1683
1684	ssh curie91 /usr/bin/ccc_myproject > $1
1685
1686	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1687	}
1688
1689	#D-#==================================================
1690	#D-function IGCM_sys_getJobSchedulerID
1691	#D-* Purpose: Get the job ID during execution
1692	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1693	#D-
1694	function IGCM_sys_getJobSchedulerID {
1695	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1696	if ( $DEBUG_sys ) ; then
1697	echo "IGCM_sys_getJobSchedulerID"
1698	fi
1699
1700	eval ${1}=${BRIDGE_MSUB_JOBID}
1701
1702	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1703	}
1704
1705	#D-#==================================================
1706	#D-function IGCM_sys_GetJobID
1707	#D-* Purpose: Get the job ID from the JobName
1708	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1709	#D-
1710	function IGCM_sys_GetJobID {
1711	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1712	if ( $DEBUG_sys ) ; then
1713	echo "IGCM_sys_GetJobID"
1714	fi
1715
1716	# With -f option, the full job name is given in the last column
1717	ID=$( ccc_mstat -f -u $2 \| \
1718	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1719
1720	eval ${3}=${ID}
1721	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1722	}
1723
1724	#D-#==================================================
1725	#D-function IGCM_sys_CountJobInQueue
1726	#D-* Purpose: Count number of users job
1727	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1728	#D-
1729	function IGCM_sys_CountJobInQueue {
1730	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1731	if ( $DEBUG_sys ) ; then
1732	echo "IGCM_sys_CountJobInQueue"
1733	fi
1734
1735	# With -f option, the full job name is given in the last column
1736	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1737
1738	eval ${2}=${NbRun}
1739
1740	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1741	}
1742
1743	#D-#==================================================
1744	#D-function IGCM_sys_ListJobInQueue
1745	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1746	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1747	#D-
1748	function IGCM_sys_ListJobInQueue {
1749	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1750	if ( $DEBUG_sys ) ; then
1751	echo "IGCM_sys_ListJobInQueue"
1752	fi
1753
1754	# With -f option, the full job name is given in the last column
1755	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1756	'( $2 == User && \
1757	$NF != /TS/ && \
1758	$NF !~ /PACK/ && \
1759	$NF !~ /REBUILD/ && \
1760	$NF !~ /pack/ ) \
1761	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1762
1763	eval set -A ${2} ${JobList[*]}
1764
1765	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1766	}
1767
1768	#D-#==================================================
1769	#D-function IGCM_sys_atlas
1770	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1771	#D-* Examples:
1772	#D-
1773	function IGCM_sys_atlas {
1774	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1775	if ( $DEBUG_sys ) ; then
1776	echo "IGCM_sys_atlas :" $@
1777	fi
1778
1779	typeset status
1780
1781	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1782	status=$?
1783	if [ ${status} -gt 0 ] ; then
1784	echo "IGCM_sys_atlas : error code ${status}"
1785	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1786	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1787	IGCM_debug_PopStack "IGCM_sys_atlas"
1788	return 1
1789	else
1790	IGCM_debug_PopStack "IGCM_sys_atlas"
1791	return 0
1792	fi
1793
1794	IGCM_debug_PopStack "IGCM_sys_atlas"
1795	}
1796
1797	#D-#==================================================
1798	#D-function IGCM_sys_rebuild_nemo
1799	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1800	#D-* Examples:
1801	#D-
1802
1803	function IGCM_sys_rebuild_nemo {
1804	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1805	if ( $DEBUG_sys ) ; then
1806	echo "IGCM_sys_rebuild_nemo :" $@
1807	fi
1808
1809	$( ccc_home -u p86ipsl )/REBUILD_NEMO/rebuild_nemo ${1} ${2}
1810
1811	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1812	}
1813

Note: See TracBrowser for help on using the repository browser.

Download in other formats: