Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1492

Last change on this file since 1492 was 1492, checked in by jgipsl, 5 years ago
Treated case MPMD MPI only (executaionType=1) at jeanzay. Note mpivars.h currently needed for this case is sourced from main job for all cases at jeanzay.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 62.0 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Jean-Zay
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=jeanzay
71	# add default project set by ins_job
72	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
73	### jobWarningDelay in seconds
74	##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=sbatch}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Access to module command
93	#====================================================
94	. /usr/share/Modules/init/ksh
95
96	#====================================================
97	# Set environment tools (ferret, nco, cdo, rebuild, ...)
98	#====================================================
99	if [ X${TaskType} = Xcomputing ] ; then
100	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
101	else
102	module purge > /dev/null 2>&1
103	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
104	#currently no atlas
105	# . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
106
107	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
108	export UVCDAT_ANONYMOUS_LOG=FALSE
109	fi
110
111	# FYI
112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
113	[ ! X${TaskType} = Xchecking ] && module list
114
115	# For AddNoise and AddPertu3DOCE
116	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
117	#jg no addnoise
118	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
119
120	# For rebuild
121	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
122	#jg no yet any rebuild
123	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
124
125	#====================================================
126	# Host specific DIRECTORIES
127	#====================================================
128
129	#====================================================
130	#- MirrorlibIGCM for frontend
131	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
132
133	#====================================================
134	#- libIGCM_POST for frontend
135	typeset -r libIGCM_POST=${libIGCM}
136
137	#====================================================
138	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
139	typeset -r R_EXE="${MODIPSL}/bin"
140
141	#====================================================
142	#- SUBMIT_DIR : submission dir
143	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
144
145	#====================================================
146	#- IN
147	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
148
149	#====================================================
150	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
151	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
152
153	#====================================================
154	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
155	typeset -r OUTCOMMAND_PATH=/tmp
156
157	#====================================================
158	#- HOST_MPIRUN_COMMAND
159	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
160
161	#====================================================
162	#- Max number of arguments passed to nco operator or demigration command
163	UNIX_MAX_LIMIT=360
164
165	#====================================================
166	#- set PackDefault to true
167	PackDefault=true
168
169	#====================================================
170	#- Default number of MPI task for IPSL coupled model
171	#- required for backward compatibility
172	#-
173	DEFAULT_NUM_PROC_OCE=5
174	DEFAULT_NUM_PROC_CPL=1
175	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
176
177	#====================================================
178	#- Number of computing cores per node
179	#-
180	NB_CORE_PER_NODE=40
181
182	#D-#==================================================
183	#D-function IGCM_sys_defineArchives
184	#D-* Purpose:
185	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
186	#D-* Define ARCHIVE : Dedicated to large files
187	#D-* Define STORAGE : Dedicated to small/medium files
188	#D-* Define R_OUT : Output tree located on ARCHIVE
189	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
190	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
191	#D-* if SpaceName=TEST everything is stored on SCRATCH
192	#D-* Examples:
193	#D-
194	function IGCM_sys_defineArchives {
195	IGCM_debug_PushStack "IGCM_sys_defineArchives"
196
197	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
198	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
199	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
200	# Default option: Change IDRPROJ according to the project used for submission
201
202	# jg following line does not work in a submitted job:
203	eval $(idrenv -d ${PROJECT})
204	else
205	# Use the project set by the variable DataProject in section UserChoices in config.card
206	eval $(idrenv -d ${config_UserChoices_DataProject})
207	IGCM_debug_Print 1 "Change DataProject red in config.card : ${config_UserChoices_DataProject}"
208	fi
209	IGCM_debug_Print 1 "Data project for output to be used: ${IDRPROJ}"
210
211	#====================================================
212	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
213	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
214
215	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
216	#====================================================
217	#- ARCHIVE (dedicated to large files)
218	ARCHIVE=${config_UserChoices_ARCHIVE}
219	else
220	#====================================================
221	#- ARCHIVE (dedicated to large files)
222	ARCHIVE=${STORE}
223	fi
224
225	if [ ! X${config_UserChoices_STORAGE} = X ]; then
226	#====================================================
227	#- STORAGE (dedicated to small/medium files)
228	STORAGE=${config_UserChoices_STORAGE}
229	else
230	#====================================================
231	#- STORAGE (dedicated to small/medium files)
232	STORAGE=${WORK}
233	fi
234
235	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
236	#====================================================
237	#- R_OUT
238	R_OUT=${SCRATCH}/IGCM_OUT
239
240	#====================================================
241	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
242	R_FIG=${SCRATCH}/IGCM_OUT
243
244	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
245
246	else
247	#====================================================
248	#- R_OUT
249	R_OUT=${ARCHIVE}/IGCM_OUT
250
251	#====================================================
252	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
253	R_FIG=${STORAGE}/IGCM_OUT
254	fi
255
256	#====================================================
257	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
258	CMIP6_BUF=${STORAGE}/IGCM_OUT
259
260	#====================================================
261	#- R_BUF (ONLY FOR double copy an scratch)
262	R_BUF=${SCRATCH}/IGCM_OUT
263
264	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
265	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
266	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
267
268	IGCM_debug_PopStack "IGCM_sys_defineArchives"
269	}
270
271	#D-#==================================================
272	#D-function IGCM_sys_RshArchive
273	#D-* Purpose: Archive rsh command
274	#D-* Examples:
275	#D-
276	function IGCM_sys_RshArchive {
277	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
278	/bin/ksh <<-EOF
279	${@}
280	EOF
281	status=$?
282	if [ ${status} -gt 0 ] ; then
283	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
284	IGCM_debug_Exit "IGCM_sys_RshArchive"
285	fi
286	IGCM_debug_PopStack "IGCM_sys_RshArchive"
287	}
288
289	#D-#==================================================
290	#D-function IGCM_sys_RshArchive_NoError
291	#D-* Purpose: Archive rsh command, without error
292	#D-* used only in monitoring.job
293	#D-* Examples:
294	#D-
295	function IGCM_sys_RshArchive_NoError {
296	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
297	/bin/ksh <<-EOF
298	${@} 2> /dev/null
299	EOF
300	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
301	}
302
303	#D-#==================================================
304	#D-function IGCM_sys_MkdirArchive
305	#D-* Purpose: Mkdir on Archive
306	#D-* Examples:
307	#D-
308	function IGCM_sys_MkdirArchive {
309	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
310	if ( $DEBUG_sys ) ; then
311	echo "IGCM_sys_MkdirArchive :" $@
312	fi
313	#- creation de repertoire sur le serveur fichier
314	if [ ! -d ${1} ]; then
315	\mkdir -p $1
316	status=$?
317
318	if [ ${status} -gt 0 ] ; then
319	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
320	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
321	fi
322	fi
323	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
324	}
325
326	#D-#==================================================
327	#D-function IGCM_sys_TestDirArchive
328	#D-* Purpose: Test Directory that must exists on Archive
329	#D-* Examples:
330	#D-
331	function IGCM_sys_TestDirArchive {
332	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
333	if ( $DEBUG_sys ) ; then
334	echo "IGCM_sys_TestDirArchive :" $@
335	fi
336	typeset ExistFlag
337	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
338	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
339	return ${ExistFlag}
340	}
341
342	#D-#==================================================
343	#D-function IGCM_sys_IsFileArchived
344	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
345	#D-* Examples:
346	#D-
347	function IGCM_sys_IsFileArchived {
348	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
349	if ( $DEBUG_sys ) ; then
350	echo "IGCM_sys_IsFileArchived :" $@
351	fi
352	typeset IsArchivedFlag
353	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
354	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
355
356	return ${IsArchivedFlag}
357	}
358
359	#D-#==================================================
360	#D-function IGCM_sys_TestFileArchive
361	#D-* Purpose: Test file that must NOT EXISTS on Archive
362	#D-* Examples:
363	#D-
364	function IGCM_sys_TestFileArchive {
365	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
366	typeset ExistFlag
367	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
368	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
369
370	return ${ExistFlag}
371	}
372
373	#D-#==================================================
374	#D-function IGCM_sys_CountFileArchive
375	#D-* Purpose: Count files on Archive filesystem
376	#D-* Examples:
377	#D-
378	function IGCM_sys_CountFileArchive {
379	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
380	ls ${@} 2>/dev/null \| wc -l
381	if [ $? -gt 0 ] ; then
382	echo "IGCM_sys_CountFileArchive : erreur."
383	fi
384	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
385	}
386
387	#D-#==================================================
388	#D-function IGCM_sys_Tree
389	#D-* Purpose: Tree directories with files on ${ARCHIVE}
390	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
391	#D-
392	function IGCM_sys_Tree {
393	IGCM_debug_PushStack "IGCM_sys_Tree" $@
394	if ( $DEBUG_sys ) ; then
395	echo "IGCM_sys_Tree :" $@
396	fi
397
398	\ls -lR ${@}
399
400	IGCM_debug_PopStack "IGCM_sys_Tree"
401	}
402
403	#D-#==================================================
404	#D-function IGCM_sys_Qsub
405	#D-* Purpose: Qsub new job
406	#D-* Examples:
407	#D-
408	function IGCM_sys_Qsub {
409	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
410	if ( $DEBUG_sys ) ; then
411	echo "IGCM_sys_Qsub :" $@
412	fi
413	typeset options status
414	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
415
416	/usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
417	status=$?
418
419	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
420	if [ ${status} -gt 0 ] ; then
421	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
422	IGCM_debug_Exit "IGCM_sys_Qsub"
423	else
424	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
425	fi
426	IGCM_debug_PopStack "IGCM_sys_Qsub"
427	}
428
429	#D-#==================================================
430	#D-function IGCM_sys_QsubPost
431	#D-* Purpose: Qsub new job on scalaire
432	#D-* Examples:
433	#D-
434	function IGCM_sys_QsubPost {
435	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
436	if ( $DEBUG_sys ) ; then
437	echo "IGCM_sys_QsubPost :" $@
438	fi
439	typeset options status
440	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
441
442	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
443	status=$?
444
445	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
446	if [ ${status} -gt 0 ] ; then
447	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
448	IGCM_debug_Exit "IGCM_sys_QsubPost"
449	else
450	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
451	fi
452	IGCM_debug_PopStack "IGCM_sys_QsubPost"
453	}
454
455	#D-*************************
456	#D- File transfer functions
457	#D-*************************
458	#D-
459
460	#D-#==================================================
461	#D-function IGCM_sys_RmRunDir
462	#D-* Purpose: rm tmpdir (dummy function most of the time batch
463	#D- scheduler will do the job)
464	#D-* Examples:
465	#D-
466	function IGCM_sys_RmRunDir {
467	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
468	if ( $DEBUG_sys ) ; then
469	echo "IGCM_sys_RmRunDir :" $@
470	fi
471
472	typeset status
473
474	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
475	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
476	status=$?
477
478	if [ ${status} -gt 0 ] ; then
479	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
480	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
481	IGCM_debug_Exit "IGCM_sys_RmRunDir"
482	else
483	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
484	fi
485	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
486	}
487
488	#D-#==================================================
489	#D-function IGCM_sys_Put_Dir
490	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
491	#D-* Examples:
492	#D-
493	function IGCM_sys_Put_Dir {
494	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
495	if ( $DEBUG_sys ) ; then
496	echo "IGCM_sys_Put_Dir :" $@
497	fi
498	if [ $DRYRUN = 0 ]; then
499	if [ ! -d ${1} ] ; then
500	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
501	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
502	return
503	fi
504
505	typeset status
506
507	# Only if we use rsync
508	#IGCM_sys_TestDirArchive $( dirname $2 )
509	#
510	#USUAL WAY
511	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
512	status=$?
513
514	if [ ${status} -gt 0 ] ; then
515	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
516	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
517	IGCM_debug_Exit "IGCM_sys_Put_Dir"
518	else
519	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
520	fi
521	fi
522	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
523	}
524
525	#D-#==================================================
526	#D-function IGCM_sys_Get_Dir
527	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
528	#D-* Examples:
529	#D-
530	function IGCM_sys_Get_Dir {
531	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
532	if ( $DEBUG_sys ) ; then
533	echo "IGCM_sys_Get_Dir :" $@
534	fi
535	if [ $DRYRUN = 0 ]; then
536	typeset NB_ESSAI DELAI status i
537	# number of tentative
538	NB_ESSAI=3
539	# time delay between tentative
540	DELAI=2
541
542	#
543	# USUAL WAY
544	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
545	ccc_hsm get -r $1
546
547	i=0
548	while [ $i -lt $NB_ESSAI ] ; do
549	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
550	status=$?
551	if [ ${status} -gt 0 ] ; then
552	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
553	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
554	sleep $DELAI
555	else
556	break
557	fi
558	(( i = i + 1 ))
559	done
560
561	if [ ${status} -gt 0 ] ; then
562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
563	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
564	IGCM_debug_Exit "IGCM_sys_Get_Dir"
565	else
566	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
567	fi
568	fi
569	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
570	}
571
572	#D-#==================================================
573	#D-function IGCM_sys_Put_Rest
574	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
575	#D- File and target directory must exist.
576	#D-* Examples:
577	#D-
578	function IGCM_sys_Put_Rest {
579	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
580	if ( $DEBUG_sys ) ; then
581	echo "IGCM_sys_Put_Rest :" $@
582	fi
583	if [ $DRYRUN = 0 ]; then
584	if [ ! -f ${1} ] ; then
585	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
586	IGCM_debug_Exit "IGCM_sys_Put_Rest"
587	fi
588
589	typeset status
590	#
591	# USUAL WAY
592	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593	status=$?
594
595	# #RSYNC WITH NETWORK SSH CALL
596	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
597	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
598
599	# #RSYNC WITH NFS USE
600	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
601	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
602
603	# status=$?
604	# IGCM_sys_Rsync_out $status
605
606	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
607	# (( status=status+$? ))
608
609	if [ ${status} -gt 0 ] ; then
610	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
611	[ -f ${1} ] && ls -l ${1}
612	[ -f ${2} ] && ls -l ${2}
613	[ -f ${2}/${1} ] && ls -l ${2}/${1}
614	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
615	IGCM_debug_Exit "IGCM_sys_Put_Rest"
616	else
617
618	if [ X${JobType} = XRUN ] ; then
619	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
620	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
621	fi
622
623	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
624	fi
625	fi
626	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
627	}
628
629	#D-#==================================================
630	#D-function IGCM_sys_Put_Out
631	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
632	#D-* Examples:
633	#D-
634	function IGCM_sys_Put_Out {
635	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
636	if ( $DEBUG_sys ) ; then
637	echo "IGCM_sys_Put_Out :" $@
638	fi
639
640	typeset NB_ESSAI DELAI status i exist skip
641	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
642
643	# number of tentative
644	NB_ESSAI=3
645	# time delay between tentative
646	DELAI=2
647
648	if [ $DRYRUN = 0 ]; then
649	if [ ! -f ${1} ] ; then
650	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
651	IGCM_debug_PopStack "IGCM_sys_Put_Out"
652	return 1
653	fi
654	#
655	IGCM_sys_MkdirArchive $( dirname $2 )
656	#
657	exist=false
658	skip=false
659	if [ -f $2 ] ; then
660	IGCM_debug_Print 1 "$2 already exist"
661	ccc_hsm get $2
662	exist=true
663	if [ "X$( diff $1 $2 )" = X ] ; then
664	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
665	skip=true
666	else
667	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
668	skip=false
669	fi
670	fi
671	#
672	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
673	IGCM_sys_Chmod u+w $2
674	fi
675
676	if [ X${skip} = Xfalse ] ; then
677	i=0
678	while [ $i -lt $NB_ESSAI ] ; do
679	# Identify file system
680	fileDeviceNumberInHex=$( stat -c %d $1 )
681	status=$?
682	if [ ${status} -gt 0 ] ; then
683	IGCM_debug_Exit "IGCM_sys_Put_Out"
684	fi
685	# Identify file system
686	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
687	status=$?
688	if [ ${status} -gt 0 ] ; then
689	IGCM_debug_Exit "IGCM_sys_Put_Out"
690	fi
691
692	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
693	# They are not on the same device. USUAL WAY
694	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
695	status=$?
696	else
697	# They are on the same device. NOT SO USUAL WAY
698	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
699	status=$?
700	fi
701	if [ ${status} -gt 0 ]; then
702	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
703	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
704	[ -f ${1} ] && ls -l ${1}
705	[ -f ${2} ] && ls -l ${2}
706	[ -f ${2}/${1} ] && ls -l ${2}/${1}
707	sleep $DELAI
708	else
709	break
710	fi
711	(( i = i + 1 ))
712	done
713	fi
714
715	# #RSYNC WITH NETWORK SSH CALL
716	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
717	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
718
719	# #RSYNC WITH NFS USE
720	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
721	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
722
723	# status=$?
724	# IGCM_sys_Rsync_out $status
725
726	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
727	# (( status=status+$? ))
728
729	if [ ${status} -gt 0 ] ; then
730	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
731	[ -f ${1} ] && ls -l ${1}
732	[ -f ${2} ] && ls -l ${2}
733	[ -f ${2}/${1} ] && ls -l ${2}/${1}
734	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
735	IGCM_debug_Exit "IGCM_sys_Put_Out"
736	else
737
738	if [ X${JobType} = XRUN ] ; then
739	if [ X${3} = X ] ; then
740	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
741	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
742	fi
743	fi
744
745	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
746	fi
747	fi
748	IGCM_debug_PopStack "IGCM_sys_Put_Out"
749	return 0
750	}
751
752	#D-#==================================================
753	#D-function IGCM_sys_Get
754	#D-* Purpose: Get a file from ${ARCHIVE}
755	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
756	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
757	function IGCM_sys_Get {
758	IGCM_debug_PushStack "IGCM_sys_Get" $@
759
760	typeset DEST dm_liste target file_work
761	typeset NB_ESSAI DELAI status i
762
763	if ( $DEBUG_sys ) ; then
764	echo "IGCM_sys_Get :" $@
765	fi
766
767	# number of tentative
768	NB_ESSAI=3
769	# time delay between tentative
770	DELAI=2
771
772	if [ $DRYRUN -le 2 ]; then
773	if [ X${1} = X'/l' ] ; then
774	eval set +A dm_liste \${${2}}
775	else
776	eval set +A dm_liste ${1}
777	fi
778	eval DEST=\${${#}}
779	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
780	status=$?
781	if [ ${status} -gt 0 ] ; then
782	echo "WARNING IGCM_sys_Get : error code ${status}"
783	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
784	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
785	fi
786
787	#if [ ${status} -gt 0 ] ; then
788	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
789	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
790	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
791	# sleep 30
792	# echo "We try another time"
793	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
794	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
795	# status=$?
796	# if [ ${status} -gt 0 ] ; then
797	# echo "ERROR IGCM_sys_Get : again demigration error :"
798	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
799	# IGCM_debug_Exit "IGCM_sys_Get"
800	# fi
801	# else
802	# echo "ERROR IGCM_sys_Get : demigration error :"
803	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
804	# IGCM_debug_Exit "IGCM_sys_Get"
805	# fi
806	#fi
807
808	# #RSYNC WITH NETWORK SSH CALL
809	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
810	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
811
812	# #RSYNC WITH NFS USE
813	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
814	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
815
816	# status=$?
817	# IGCM_sys_Rsync_out $status
818
819	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
820	# (( status=status+$? ))
821
822	#USUAL WAY
823	if [ X${1} = X'/l' ] ; then
824	for target in ${dm_liste[*]} ; do
825	local_file=$( basename ${target} )
826	# test if the target file is present before the loop
827	IGCM_sys_TestFileArchive ${target}
828	status=$?
829	if [ ${status} -gt 0 ] ; then
830	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
831	IGCM_debug_Exit "IGCM_sys_Get"
832	else
833	i=0
834	while [ $i -lt $NB_ESSAI ] ; do
835	#if [ X${DoLink} = Xtrue ] ; then
836	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
837	# status=$?
838	# else
839	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
840	# status=$?
841	#fi
842	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
843	status=$?
844	if [ ${status} -gt 0 ]; then
845	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
846	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
847	sleep $DELAI
848	else
849	break
850	fi
851	(( i = i + 1 ))
852	done
853	if [ ${status} -gt 0 ] ; then
854	echo "IGCM_sys_Get : error"
855	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
856	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
857	IGCM_debug_Exit "IGCM_sys_Get"
858	else
859	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
860	fi
861	fi
862	done
863	else
864	i=0
865	while [ $i -lt $NB_ESSAI ] ; do
866	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
867	status=$?
868	if [ ${status} -gt 0 ]; then
869	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
870	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
871	sleep $DELAI
872	else
873	break
874	fi
875	(( i = i + 1 ))
876	done
877	if [ ${status} -gt 0 ] ; then
878	echo "IGCM_sys_Get : error"
879	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
880	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
881	IGCM_debug_Exit "IGCM_sys_Get"
882	else
883	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
884	fi
885	fi
886	fi
887	IGCM_debug_PopStack "IGCM_sys_Get"
888	}
889
890	#D-#==================================================
891	#D-function IGCM_sys_GetDate_Monitoring
892	#D-* Purpose: get the last year for which the monitoring has been computed
893	#D-* Examples:
894	#D-
895	function IGCM_sys_GetDate_Monitoring {
896	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
897	if ( $DEBUG_sys ) ; then
898	echo "IGCM_sys_GetDate_Monitoring :" $@
899	fi
900
901	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
902
903	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
904	}
905
906	#D-#==================================================
907	#D-function IGCM_sys_Dods_Rm
908	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
909	#D-* Examples:
910	#D-
911	function IGCM_sys_Dods_Rm {
912	if ( $DEBUG_sys ) ; then
913	echo "IGCM_sys_Dods_Rm :" $@
914	fi
915	typeset status
916	if [ $DRYRUN = 0 ]; then
917
918	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
919	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
920	# echo "Nothing has been done."
921	# return
922	# fi
923
924	if [ "$#" -eq 1 ]; then
925	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
926	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
927	status=$?
928	else
929	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
930	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
931	status=$?
932	fi
933
934	# if [ ${status} -gt 0 ] ; then
935	# echo "IGCM_sys_Dods_Rm : error."
936	# cat out_dods_rm
937	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
938	# else
939	# rm out_dods_rm
940	# fi
941
942	fi
943	return $status
944	}
945
946	#D-#==================================================
947	#D-function IGCM_sys_Dods_Cp
948	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
949	#D-* Examples:
950	#D-
951	function IGCM_sys_Dods_Cp {
952	if ( $DEBUG_sys ) ; then
953	echo "IGCM_sys_Dods_Cp :" $@
954	fi
955	typeset status
956	if [ $DRYRUN = 0 ]; then
957
958	# if [ ! -d ${R_SAVE}/${1} ] ; then
959	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
960	# echo "Nothing has been done."
961	# return
962	# fi
963
964	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
965	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
966	status=$?
967
968	# if [ ${status} -gt 0 ] ; then
969	# echo "IGCM_sys_Dods_Cp : error."
970	# cat out_dods_cp
971	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
972	# else
973	# rm out_dods_cp
974	# fi
975
976	fi
977	return $status
978	}
979
980	#D-#==================================================
981	#D-function IGCM_sys_Put_Dods
982	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
983	#D-* Examples:
984	#D-
985	function IGCM_sys_Put_Dods {
986	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
987	if ( $DEBUG_sys ) ; then
988	echo "IGCM_sys_Put_Dods :" $@
989	fi
990	typeset status
991	if [ $DRYRUN = 0 ]; then
992	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
993	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
994	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
995	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
996	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
997	return
998	fi
999
1000	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1001	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1002	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1003	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1004	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1005	return
1006	fi
1007	#
1008	if [ -d ${R_SAVE}/${1} ] ; then
1009	cd ${R_SAVE}
1010	elif [ -d ${R_FIGR}/${1} ] ; then
1011	cd ${R_FIGR}
1012	fi
1013
1014	IGCM_sys_Dods_Rm ${1}
1015	IGCM_sys_Dods_Cp ${1}
1016	status=0
1017
1018	if [ ${status} -gt 0 ] ; then
1019	echo "IGCM_sys_Put_Dods : error."
1020	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1021	fi
1022	fi
1023	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1024	}
1025
1026	##############################################################
1027	# REBUILD OPERATOR
1028
1029	#D-#==================================================
1030	#D-function IGCM_sys_sync
1031	#D-* Purpose: flush buffer on disk
1032	#D-* Examples:
1033	#D-
1034	function IGCM_sys_sync {
1035	IGCM_debug_PushStack "IGCM_sys_sync" $@
1036	if ( $DEBUG_sys ) ; then
1037	echo "IGCM_sys_sync :" $@
1038	fi
1039
1040	/bin/sync
1041
1042	IGCM_debug_PopStack "IGCM_sys_sync"
1043	}
1044
1045	############################################################
1046	# Activate Running Environnment Variables
1047
1048	#D-#==================================================
1049	#D-function IGCM_sys_activ_variables
1050	#D-* Purpose: set environement variables prior to execution
1051	#D-* Examples:
1052	#D-
1053	function IGCM_sys_activ_variables {
1054	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1055	if ( $DEBUG_sys ) ; then
1056	echo "IGCM_sys_activ_variables"
1057	fi
1058
1059	# --------------------------------------------------------------------
1060	#D- MODULE specifications
1061	# --------------------------------------------------------------------
1062	# Source the file EnvFile containing all module load needed to run the model.
1063	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1064	# it will be used. Otherwise a default file will be used.
1065	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1066	# EnvFile is set in config.card and the file exists
1067	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1068	EnvFile=${config_UserChoices_EnvFile}
1069	else
1070	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1071	module purge > /dev/null 2>&1
1072	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1073	fi
1074	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1075	IGCM_debug_Print 1 ${EnvFile}
1076	. ${EnvFile}
1077
1078	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1079	module list
1080
1081	# --------------------------------------------------------------------
1082	#D- MPI specifications
1083	# --------------------------------------------------------------------
1084
1085	# --------------------------------------------------------------------
1086	#D- Other specifications
1087	# --------------------------------------------------------------------
1088
1089	ulimit -s unlimited
1090
1091	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1092	}
1093
1094	############################################################
1095	# Desactivate Running Environnment Variables
1096
1097	#D-#==================================================
1098	#D-function IGCM_sys_desactiv_variables
1099	#D-* Purpose: unset environement variables after execution
1100	#D-* Examples:
1101	#D-
1102	function IGCM_sys_desactiv_variables {
1103	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1104	if ( $DEBUG_sys ) ; then
1105	echo "IGCM_sys_desactiv_variables"
1106	fi
1107	# --------------------------------------------------------------------
1108	#D- MPI specifications
1109	# --------------------------------------------------------------------
1110
1111	# --------------------------------------------------------------------
1112	#D- Other specifications
1113	# --------------------------------------------------------------------
1114
1115	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1116	}
1117
1118	############################################################
1119	# Update job headers to be used by the scheduler
1120
1121	#D-#==================================================
1122	#D-function IGCM_sys_updateHeaders
1123	#D-* Purpose: Update job headers to be used by the scheduler
1124	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1125	#D-
1126	function IGCM_sys_updateHeaders {
1127	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1128	if ( $DEBUG_sys ) ; then
1129	echo "IGCM_sys_updateHeaders"
1130	fi
1131	typeset file
1132	file=$1
1133
1134	if [ ${executionType} -eq 1 ] ; then
1135	# MPMD + MPI
1136	sed -e "/::openMPthreads::/d" \
1137	-e "s/::JobNumProcTot::/${coreNumber}/" \
1138	-e "/#MSUB -x/d" \
1139	-e "/--cpu_bind=none/d" \
1140	${file} > ${file}.tmp
1141
1142	elif [ ${executionType} -eq 2 ] ; then
1143	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1144	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1145	sed -e "/::openMPthreads::/d" \
1146	-e "s/::JobNumProcTot::/${coreNumber}/" \
1147	${file} > ${file}.tmp
1148	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1149	sed -e "/::openMPthreads::/d" \
1150	-e "s/::JobNumProcTot::/${coreNumber}/" \
1151	-e "/--cpu_bind=none/d" \
1152	${file} > ${file}.tmp
1153	else
1154	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1155	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1156	fi
1157	elif [ ${executionType} -eq 3 ] ; then
1158	# SPMD + MPI/OMP
1159	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1160	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1161	-e "/#MSUB -x/d" \
1162	-e "/--cpu_bind=none/d" \
1163	${file} > ${file}.tmp
1164
1165	elif [ ${executionType} -eq 4 ] ; then
1166	# SPMD + MPI only
1167	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1168	-e "/::openMPthreads::/d" \
1169	-e "/#MSUB -x/d" \
1170	-e "/--cpu_bind=none/d" \
1171	${file} > ${file}.tmp
1172
1173	elif [ ${executionType} -eq 5 ] ; then
1174	# SPMD + OMP only
1175	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1176	-e "/::JobNumProcTot::/d" \
1177	-e "/#MSUB -x/d" \
1178	-e "/--cpu_bind=none/d" \
1179	${file} > ${file}.tmp
1180
1181	elif [ ${executionType} -eq 6 ] ; then
1182	# SEQUENTIAL THEN
1183	sed -e "s/::JobNumProcTot::/1/" \
1184	-e "/::openMPthreads::/d" \
1185	-e "/#MSUB -x/d" \
1186	-e "/--cpu_bind=none/d" \
1187	${file} > ${file}.tmp
1188
1189	fi
1190
1191	IGCM_sys_Mv ${file}.tmp ${file}
1192
1193	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1194	}
1195
1196	############################################################
1197	# Build MPI/OMP scripts run file (dummy function)
1198
1199	#D-#==================================================
1200	#D-function IGCM_sys_build_run_file
1201	#D-* Purpose: build run file (deprecated)
1202	#D-* Examples:
1203	#D-
1204	function IGCM_sys_build_run_file {
1205	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1206	}
1207
1208	############################################################
1209	# Build MPI/OMP scripts
1210
1211	#D-#==================================================
1212	#D-function IGCM_sys_build_execution_scripts
1213	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1214	#D-* Examples:
1215	#D-
1216	function IGCM_sys_build_execution_scripts
1217	{
1218	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1219	if ( $DEBUG_sys ) ; then
1220	echo "IGCM_sys_build_execution_scripts " $@
1221	fi
1222
1223	EXECUTION=${HOST_MPIRUN_COMMAND}
1224
1225	if ( ${OK_PARA_MPMD} ) ; then
1226
1227	# MPMD mode
1228	# 1 MPI only
1229	# 2 MPI/OpenMP mpirun method
1230	# 3 MPI/OpenMP ccc_mprun method
1231
1232	if [ -f run_file ] ; then
1233	IGCM_sys_Rm -f run_file
1234	fi
1235	touch run_file
1236
1237	# case 1 : Only MPI (MPMD)
1238	if ( ! ${OK_PARA_OMP} ) ; then
1239	# Build run_file
1240	current_core=0
1241	# First loop on the components for the coupler ie oasis (only if oasis3)
1242	# the coupler ie oasis3 must be the first one
1243	for comp in ${config_ListOfComponents[*]} ; do
1244
1245	eval ExeNameIn=\${config_Executable_${comp}[0]}
1246	eval ExeNameOut=\${config_Executable_${comp}[1]}
1247
1248	# Only if the component has an executable
1249	if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1250
1251	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1252	(( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
1253	echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
1254	(( current_core = ${end_core} + 1 ))
1255	fi
1256	done
1257
1258	EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
1259
1260	IGCM_sys_Chmod u+x run_file
1261	if ( $DEBUG_sys ) ; then
1262	echo "run_file contains : "
1263	cat run_file
1264	fi
1265
1266	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1267
1268	# 2 MPI/OpenMP mpirun method
1269
1270	# Use of mpirun instead of ccc_mprun
1271	EXECUTION="time mpirun"
1272
1273	# Hosts treatment
1274	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1275
1276	i=0
1277	rm -f hosts rankfile
1278	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1279	for nodes in `cat hosts.tmp`
1280	do
1281	host[$i]=$nodes
1282	echo "${host[$i]}" >> hosts
1283	IGCM_debug_Print 1 ${host[$i]}
1284	i=$((i+1))
1285	done
1286	rm -f hosts.tmp
1287
1288	listnodes=${host[*]}
1289
1290	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1291
1292	# Initialisation
1293	rank=0
1294	current_core=0
1295	init_exec=n
1296
1297	# Loop on the components
1298	for comp in ${config_ListOfComponents[*]} ; do
1299
1300	eval ExeNameIn=\${config_Executable_${comp}[0]}
1301	eval ExeNameOut=\${config_Executable_${comp}[1]}
1302
1303	# Not possible if oasis has an executable (i.e old version of oasis3)
1304	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1305	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1306	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1307	IGCM_debug_Verif_Exit
1308	fi
1309
1310	# Only if we really have an executable for the component :
1311	if [ "X${ExeNameOut}" != X\"\" ] ; then
1312
1313	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1314	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1315	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1316
1317	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1318	echo "" >> script_${ExeNameOut}.ksh
1319	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1320
1321	# Check if the number of threads is correct
1322	case ${comp_proc_omp_loc} in
1323	2\|4\|8\|16)
1324	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1325	;;
1326	*)
1327	IGCM_debug_Exit "ERROR with OMP parameters !"
1328	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1329	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1330	IGCM_debug_Verif_Exit
1331	;;
1332	esac
1333	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1334	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1335	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1336	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1337	fi
1338
1339	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1340	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1341	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1342	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1343	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1344
1345	if [ ${init_exec} = y ] ; then
1346	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1347	else
1348	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1349	init_exec=y
1350	fi
1351
1352	# Build rankfile : method used to assign cores and nodes for the MPI process
1353	# Ex :
1354	#rank 0=curie5296 slot=0,1,2,3
1355	#rank 1=curie5296 slot=4,5,6,7
1356	# Example of final command :
1357	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1358	# with script_lmdz.x.ksh :
1359	# #!/bin/ksh
1360	#export KMP_STACKSIZE=3g
1361	#export KMP_LIBRARY=turnaround
1362	#export MKL_SERIAL=YES
1363	#OMP_NUM_THREADS=4
1364	#./lmdz.x
1365	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1366	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1367	else
1368	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1369	fi
1370	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1371	(( index_host = current_core / NB_CORE_PER_NODE ))
1372	host_value=${host[${index_host}]}
1373	(( slot = current_core % NB_CORE_PER_NODE ))
1374	virg=","
1375	string_final=""
1376	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1377	string=$index$virg
1378	string_final=$string_final$string
1379	done
1380	string_final=$( echo $string_final \| sed "s/.$//" )
1381	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1382	(( rank = rank + 1 ))
1383	(( current_core = current_core + offset_comp_proc_loc ))
1384	done
1385	fi
1386	done
1387
1388	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1389
1390	# 3 MPI/OpenMP ccc_mprun method
1391
1392	# MPI-OpenMP (MPMD)
1393	# example of run_file
1394	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1395	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1396	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1397
1398	# Loop on the components
1399	for comp in ${config_ListOfComponents[*]} ; do
1400
1401	eval ExeNameIn=\${config_Executable_${comp}[0]}
1402	eval ExeNameOut=\${config_Executable_${comp}[1]}
1403
1404	# Not possible if oasis has an executable (i.e old version of oasis3)
1405	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1406	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1407	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1408	IGCM_debug_Verif_Exit
1409	fi
1410
1411	# Only if we really have an executable for the component :
1412	if [ "X${ExeNameOut}" != X\"\" ] ; then
1413
1414	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1415	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1416
1417	# Build script files
1418
1419	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1420	echo "" >> script_${ExeNameOut}.ksh
1421	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1422
1423	# Check if the number of threads is correct
1424	case ${comp_proc_omp_loc} in
1425	2\|4\|8\|12\|16\|24)
1426	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1427	;;
1428	*)
1429	IGCM_debug_Exit "ERROR with OMP parameters !"
1430	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1431	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1432	IGCM_debug_Verif_Exit
1433	;;
1434	esac
1435	### only for LMDZ?
1436	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1437	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1438	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1439	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1440
1441	fi
1442
1443	# to have out/err per process on different files
1444	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1445	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1446
1447	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1448
1449	# Complete run_file
1450
1451	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1452
1453	fi
1454	done
1455
1456	## variable added to stop after 60s instead of 600s by default.
1457	## This is used when no error comes from executables and when something stopped an executable without notice.
1458	export SLURM_WAIT=60
1459
1460	## module advised by TGCC (instead of 2 variables)
1461	module load feature/bridge/heterogenous_mpmd
1462
1463	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1464
1465	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1466
1467	IGCM_sys_Chmod u+x run_file
1468	if ( $DEBUG_sys ) ; then
1469	echo "run_file contains : "
1470	cat run_file
1471	fi
1472
1473	else
1474
1475	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1476
1477	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1478
1479	else
1480	# Only one executable (SPMD mode).
1481
1482	for comp in ${config_ListOfComponents[*]} ; do
1483
1484	# Only if we really have an executable for the component :
1485	eval ExeNameOut=\${config_Executable_${comp}[1]}
1486	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1487
1488	# Build script files
1489
1490	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1491	echo "" >> script_${ExeNameOut}.ksh
1492	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1493
1494	if ( ${OK_PARA_OMP} ) ; then
1495	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1496	echo "" >> script_${ExeNameOut}.ksh
1497	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1498	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1499	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1500	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1501	fi
1502
1503	if ( ${OK_PARA_MPI} ) ; then
1504	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1505	# Default : ccc_mprun used if nb_proc gt 1
1506	# to have out/err per process on different files
1507	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1508	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1509	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1510	else
1511	# Default : ccc_mprun is NOT used if nb_proc eq 1
1512	# to have out/err per process on different files
1513	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1514	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1515	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1516	fi
1517
1518	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1519	cat script_${ExeNameOut}.ksh
1520
1521	fi
1522
1523	done
1524
1525	fi # ${OK_PARA_MPMD}
1526
1527	IGCM_debug_Print 1 "sys Irene : execution command is "
1528	IGCM_debug_Print 1 "$EXECUTION"
1529
1530	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1531	}
1532
1533	#D-#==================================================
1534	#D-function IGCM_sys_check_path
1535	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1536	#D-* do not point to an important use directory. Stop immediately in that case.
1537	#D-* Examples:
1538	#D-
1539	function IGCM_sys_check_path {
1540	IGCM_debug_PushStack "IGCM_sys_check_path"
1541	if ( $DEBUG_sys ) ; then
1542	echo "IGCM_sys_check_path"
1543	fi
1544
1545	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1546	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1547	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1548	IGCM_debug_Exit "This will stop the job"
1549	fi
1550	IGCM_debug_PopStack "IGCM_sys_check_path"
1551	}
1552
1553	#D-#==================================================
1554	#D-function IGCM_sys_check_quota
1555	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1556	#D-* Examples:
1557	#D-
1558	function IGCM_sys_check_quota {
1559	IGCM_debug_PushStack "IGCM_sys_check_quota"
1560	if ( $DEBUG_sys ) ; then
1561	echo "IGCM_sys_check_quota"
1562	fi
1563	# Limit of quota (in %)
1564	limit_quota=90
1565
1566	# Check of the volume
1567	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1568	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1569
1570	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1571
1572	unit_avail=${volume_avail: -1}
1573	unit_quota=${volume_quota: -1}
1574
1575	if [ "${unit_quota}" = "*" ] ; then
1576	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1577	IGCM_debug_Print 1 "More than 100% of your quota is used"
1578	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1579	IGCM_debug_Print 1 "You must have more than 10% available to run"
1580	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1581	IGCM_debug_Verif_Exit
1582	fi
1583
1584	temp_avail=${volume_avail%%${volume_avail: -1}*}
1585	temp_quota=${volume_quota%%${volume_quota: -1}*}
1586
1587	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1588
1589	# Convertion
1590	if [ ${volume_avail: -1} = "T" ] ; then
1591	(( temp_avail = temp_avail * 1000000000000 ))
1592	elif [ ${volume_avail: -1} = "G" ] ; then
1593	(( temp_avail = temp_avail * 1000000000 ))
1594	elif [ ${volume_avail: -1} = "M" ] ; then
1595	(( temp_avail = temp_avail * 1000000 ))
1596	elif [ ${volume_avail: -1} = "k" ] ; then
1597	(( temp_avail = temp_avail * 1000 ))
1598	else
1599	(( temp_avail = volume_avail ))
1600	fi
1601	if [ ${volume_quota: -1} = "T" ] ; then
1602	(( temp_quota = temp_quota * 1000000000000 ))
1603	elif [ ${volume_quota: -1} = "G" ] ; then
1604	(( temp_quota = temp_quota * 1000000000 ))
1605	elif [ ${volume_quota: -1} = "M" ] ; then
1606	(( temp_quota = temp_quota * 1000000 ))
1607	elif [ ${volume_quota: -1} = "k" ] ; then
1608	(( temp_quota = temp_quota * 1000 ))
1609	else
1610	(( temp_quota = volume_quota ))
1611	fi
1612	fi
1613
1614	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1615	# echo "volume ratio is " $quota_volume
1616
1617	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1618	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1619	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1620	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1621	IGCM_debug_Print 1 "You must have more than 10% available to run"
1622	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1623	IGCM_debug_Verif_Exit
1624	fi
1625
1626	fi
1627
1628	# Check of the number of inodes
1629
1630	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1631	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1632
1633	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1634
1635	unit_avail=${inode_avail: -1}
1636	unit_quota=${inode_quota: -1}
1637
1638	if [ "${unit_quota}" = "*" ] ; then
1639	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1640	IGCM_debug_Print 1 "More than 100% of your quota is used"
1641	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1642	IGCM_debug_Print 1 "You must have more than 10% available to run"
1643	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1644	IGCM_debug_Verif_Exit
1645	fi
1646
1647	temp_avail=${inode_avail%%${inode_avail: -1}*}
1648	temp_quota=${inode_quota%%${inode_quota: -1}*}
1649
1650	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1651
1652	# Convertion
1653	if [ ${inode_avail: -1} = "T" ] ; then
1654	(( temp_avail = temp_avail * 1000000000000 ))
1655	elif [ ${inode_avail: -1} = "G" ] ; then
1656	(( temp_avail = temp_avail * 1000000000 ))
1657	elif [ ${inode_avail: -1} = "M" ] ; then
1658	(( temp_avail = temp_avail * 1000000 ))
1659	elif [ ${inode_avail: -1} = "k" ] ; then
1660	(( temp_avail = temp_avail * 1000 ))
1661	else
1662	(( temp_avail = inode_avail ))
1663	fi
1664
1665	if [ ${inode_quota: -1} = "T" ] ; then
1666	(( temp_quota = temp_quota * 1000000000000 ))
1667	elif [ ${inode_quota: -1} = "G" ] ; then
1668	(( temp_quota = temp_quota * 1000000000 ))
1669	elif [ ${inode_quota: -1} = "M" ] ; then
1670	(( temp_quota = temp_quota * 1000000 ))
1671	elif [ ${inode_quota: -1} = "k" ] ; then
1672	(( temp_quota = temp_quota * 1000 ))
1673	else
1674	(( temp_quota = inode_quota ))
1675	fi
1676	fi
1677	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1678	# echo "inode ratio is " $quota_inode
1679
1680	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1681	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1682	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1683	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1684	IGCM_debug_Print 1 "You must have more than 10% available to run"
1685	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1686	IGCM_debug_Verif_Exit
1687	fi
1688	fi
1689	IGCM_debug_PopStack "IGCM_sys_check_quota"
1690	}
1691
1692	#D-#==================================================
1693	#D-function IGCM_sys_projectAccounting
1694	#D-* Purpose: store project accounting information in a file
1695	#D-* Examples:
1696	#D-
1697	function IGCM_sys_projectAccounting {
1698	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1699	if ( $DEBUG_sys ) ; then
1700	echo "IGCM_sys_check_quota"
1701	fi
1702
1703	ssh irene191 /usr/bin/ccc_myproject > $1
1704
1705	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1706	}
1707
1708	#D-#==================================================
1709	#D-function IGCM_sys_getJobSchedulerID
1710	#D-* Purpose: Get the job ID during execution
1711	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1712	#D-
1713	function IGCM_sys_getJobSchedulerID {
1714	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1715	if ( $DEBUG_sys ) ; then
1716	echo "IGCM_sys_getJobSchedulerID"
1717	fi
1718
1719	eval ${1}=${BRIDGE_MSUB_JOBID}
1720
1721	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1722	}
1723
1724	#D-#==================================================
1725	#D-function IGCM_sys_GetJobID
1726	#D-* Purpose: Get the job ID from the JobName
1727	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1728	#D-
1729	function IGCM_sys_GetJobID {
1730	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1731	if ( $DEBUG_sys ) ; then
1732	echo "IGCM_sys_GetJobID"
1733	fi
1734
1735	# With -f option, the full job name is given in the last column
1736	ID=$( ccc_mstat -f -u $2 \| \
1737	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1738
1739	eval ${3}=${ID}
1740	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1741	}
1742
1743	#D-#==================================================
1744	#D-function IGCM_sys_CountJobInQueue
1745	#D-* Purpose: Count number of users job
1746	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1747	#D-
1748	function IGCM_sys_CountJobInQueue {
1749	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1750	if ( $DEBUG_sys ) ; then
1751	echo "IGCM_sys_CountJobInQueue"
1752	fi
1753
1754	# With -f option, the full job name is given in the last column
1755	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1756
1757	eval ${2}=${NbRun}
1758
1759	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1760	}
1761
1762	#D-#==================================================
1763	#D-function IGCM_sys_ListJobInQueue
1764	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1765	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1766	#D-
1767	function IGCM_sys_ListJobInQueue {
1768	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1769	if ( $DEBUG_sys ) ; then
1770	echo "IGCM_sys_ListJobInQueue"
1771	fi
1772
1773	# With -f option, the full job name is given in the last column
1774	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1775	'( $2 == User && \
1776	$NF != /TS/ && \
1777	$NF !~ /PACK/ && \
1778	$NF !~ /REBUILD/ && \
1779	$NF !~ /pack/ ) \
1780	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1781
1782	eval set -A ${2} ${JobList[*]}
1783
1784	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1785	}
1786
1787	#D-#==================================================
1788	#D-function IGCM_sys_atlas
1789	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1790	#D-* Examples:
1791	#D-
1792	function IGCM_sys_atlas {
1793	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1794	if ( $DEBUG_sys ) ; then
1795	echo "IGCM_sys_atlas :" $@
1796	fi
1797
1798	typeset status
1799
1800	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1801	status=$?
1802	if [ ${status} -gt 0 ] ; then
1803	echo "IGCM_sys_atlas : error code ${status}"
1804	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1805	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1806	IGCM_debug_PopStack "IGCM_sys_atlas"
1807	return 1
1808	else
1809	IGCM_debug_PopStack "IGCM_sys_atlas"
1810	return 0
1811	fi
1812
1813	IGCM_debug_PopStack "IGCM_sys_atlas"
1814	}
1815
1816	#D-#==================================================
1817	#D-function IGCM_sys_rebuild_nemo
1818	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1819	#D-* Examples:
1820	#D-
1821
1822	function IGCM_sys_rebuild_nemo {
1823	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1824	if ( $DEBUG_sys ) ; then
1825	echo "IGCM_sys_rebuild_nemo :" $@
1826	fi
1827
1828	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1829
1830	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1831	}
1832

Note: See TracBrowser for help on using the repository browser.

Download in other formats: