Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1511

Last change on this file since 1511 was 1511, checked in by acosce, 4 years ago
add possibility to overwrite R_OUT and R_BUF by general variables define directly in the job - as RUN_DIR_PATH. This add is done to prevent a problem during the training day at idris wq
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 57.8 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Jean-Zay
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=jeanzay
71	# add default project set by ins_job
72	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
73	### jobWarningDelay in seconds
74	##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=sbatch}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Access to module command
93	#====================================================
94	. /usr/share/Modules/init/ksh
95
96	#====================================================
97	# Set environment tools (ferret, nco, cdo, rebuild, ...)
98	#====================================================
99	if [ X${TaskType} = Xcomputing ] ; then
100	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
101	else
102	module purge > /dev/null 2>&1
103	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
104
105	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
106
107	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
108	export UVCDAT_ANONYMOUS_LOG=FALSE
109	fi
110
111	# FYI
112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
113	[ ! X${TaskType} = Xchecking ] && module list
114
115	# For AddNoise (not yet AddPertu3DOCE)
116	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/AddNoise/bin
117
118	# For rebuild
119	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
120
121	#====================================================
122	# Host specific DIRECTORIES
123	#====================================================
124
125	#====================================================
126	#- MirrorlibIGCM for frontend
127	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
128
129	#====================================================
130	#- libIGCM_POST for frontend
131	typeset -r libIGCM_POST=${libIGCM}
132
133	#====================================================
134	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
135	typeset -r R_EXE="${MODIPSL}/bin"
136
137	#====================================================
138	#- SUBMIT_DIR : submission dir
139	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
140
141	#====================================================
142	#- IN
143	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
144
145	#====================================================
146	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
147	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
148
149	#====================================================
150	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
151	typeset -r OUTCOMMAND_PATH=/tmp
152
153	#====================================================
154	#- HOST_MPIRUN_COMMAND
155	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
156
157	#====================================================
158	#- Max number of arguments passed to nco operator or demigration command
159	UNIX_MAX_LIMIT=360
160
161	#====================================================
162	#- set PackDefault to true
163	PackDefault=true
164
165	#====================================================
166	#- Default number of MPI task for IPSL coupled model
167	#- required for backward compatibility
168	#-
169	DEFAULT_NUM_PROC_OCE=5
170	DEFAULT_NUM_PROC_CPL=1
171	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
172
173	#====================================================
174	#- Number of computing cores per node
175	#-
176	NB_CORE_PER_NODE=40
177
178	#D-#==================================================
179	#D-function IGCM_sys_defineArchives
180	#D-* Purpose:
181	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
182	#D-* Define ARCHIVE : Dedicated to large files
183	#D-* Define STORAGE : Dedicated to small/medium files
184	#D-* Define R_OUT : Output tree located on ARCHIVE
185	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
186	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
187	#D-* if SpaceName=TEST everything is stored on SCRATCH
188	#D-* Examples:
189	#D-
190	function IGCM_sys_defineArchives {
191	IGCM_debug_PushStack "IGCM_sys_defineArchives"
192
193	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
194	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
195	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
196	# Default option: Set DataProject according to the project used for submission
197	DataProject=${PROJECT}
198	else
199	# Use the project set by the variable DataProject in section UserChoices in config.card
200	IGCM_debug_Print 1 "Use DataProject as set in config.card : ${config_UserChoices_DataProject}"
201	DataProject=${config_UserChoices_DataProject}
202	fi
203	IGCM_debug_Print 1 "Data project for output to be used: ${DataProject}"
204
205	#====================================================
206	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
207	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/gpfsscratch/rech/${DataProject}/${LOGIN}/RUN_DIR/${SLURM_JOBID}_${$}}
208
209	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
210	#====================================================
211	#- ARCHIVE (dedicated to large files)
212	ARCHIVE=${config_UserChoices_ARCHIVE}
213	else
214	#====================================================
215	#- ARCHIVE (dedicated to large files)
216	ARCHIVE=/gpfsstore/rech/${DataProject}/${LOGIN}
217	fi
218
219	if [ ! X${config_UserChoices_STORAGE} = X ]; then
220	#====================================================
221	#- STORAGE (dedicated to small/medium files)
222	STORAGE=${config_UserChoices_STORAGE}
223	else
224	#====================================================
225	#- STORAGE (dedicated to small/medium files)
226	STORAGE=/gpfswork/rech/${DataProject}/${LOGIN}
227	fi
228
229	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
230	#====================================================
231	#- R_OUT
232	R_OUT=${R_OUT:=/gpfsscratch/rech/${DataProject}/${LOGIN}}/IGCM_OUT
233
234	#====================================================
235	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
236	R_FIG=${R_OUT}
237
238	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
239
240	else
241	#====================================================
242	#- R_OUT
243	R_OUT=${ARCHIVE}/IGCM_OUT
244
245	#====================================================
246	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
247	R_FIG=${STORAGE}/IGCM_OUT
248	fi
249
250	#====================================================
251	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
252	CMIP6_BUF=${STORAGE}/IGCM_OUT
253
254	#====================================================
255	#- R_BUF (ONLY FOR double copy an scratch)
256	R_BUF=${R_BUF:=/gpfsscratch/rech/${DataProject}/${LOGIN}}/IGCM_OUT
257
258	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
259	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
260	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
261
262	IGCM_debug_PopStack "IGCM_sys_defineArchives"
263	}
264
265	#D-#==================================================
266	#D-function IGCM_sys_RshArchive
267	#D-* Purpose: Archive rsh command
268	#D-* Examples:
269	#D-
270	function IGCM_sys_RshArchive {
271	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
272	/bin/ksh <<-EOF
273	${@}
274	EOF
275	status=$?
276	if [ ${status} -gt 0 ] ; then
277	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
278	IGCM_debug_Exit "IGCM_sys_RshArchive"
279	fi
280	IGCM_debug_PopStack "IGCM_sys_RshArchive"
281	}
282
283	#D-#==================================================
284	#D-function IGCM_sys_RshArchive_NoError
285	#D-* Purpose: Archive rsh command, without error
286	#D-* used only in monitoring.job
287	#D-* Examples:
288	#D-
289	function IGCM_sys_RshArchive_NoError {
290	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
291	/bin/ksh <<-EOF
292	${@} 2> /dev/null
293	EOF
294	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
295	}
296
297	#D-#==================================================
298	#D-function IGCM_sys_MkdirArchive
299	#D-* Purpose: Mkdir on Archive
300	#D-* Examples:
301	#D-
302	function IGCM_sys_MkdirArchive {
303	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
304	if ( $DEBUG_sys ) ; then
305	echo "IGCM_sys_MkdirArchive :" $@
306	fi
307	#- creation de repertoire sur le serveur fichier
308	if [ ! -d ${1} ]; then
309	\mkdir -p $1
310	status=$?
311
312	if [ ${status} -gt 0 ] ; then
313	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
314	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
315	fi
316	fi
317	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
318	}
319
320	#D-#==================================================
321	#D-function IGCM_sys_TestDirArchive
322	#D-* Purpose: Test Directory that must exists on Archive
323	#D-* Examples:
324	#D-
325	function IGCM_sys_TestDirArchive {
326	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
327	if ( $DEBUG_sys ) ; then
328	echo "IGCM_sys_TestDirArchive :" $@
329	fi
330	typeset ExistFlag
331	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
332	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
333	return ${ExistFlag}
334	}
335
336	#D-#==================================================
337	#D-function IGCM_sys_IsFileArchived
338	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
339	#D-* Examples:
340	#D-
341	function IGCM_sys_IsFileArchived {
342	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
343	if ( $DEBUG_sys ) ; then
344	echo "IGCM_sys_IsFileArchived :" $@
345	fi
346	typeset IsArchivedFlag
347	# To be implemented on Jean Zay
348	# IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
349	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
350
351	return ${IsArchivedFlag}
352	}
353
354	#D-#==================================================
355	#D-function IGCM_sys_TestFileArchive
356	#D-* Purpose: Test file that must NOT EXISTS on Archive
357	#D-* Examples:
358	#D-
359	function IGCM_sys_TestFileArchive {
360	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
361	typeset ExistFlag
362	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
363	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
364
365	return ${ExistFlag}
366	}
367
368	#D-#==================================================
369	#D-function IGCM_sys_CountFileArchive
370	#D-* Purpose: Count files on Archive filesystem
371	#D-* Examples:
372	#D-
373	function IGCM_sys_CountFileArchive {
374	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
375	ls ${@} 2>/dev/null \| wc -l
376	if [ $? -gt 0 ] ; then
377	echo "IGCM_sys_CountFileArchive : erreur."
378	fi
379	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
380	}
381
382	#D-#==================================================
383	#D-function IGCM_sys_Tree
384	#D-* Purpose: Tree directories with files on ${ARCHIVE}
385	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
386	#D-
387	function IGCM_sys_Tree {
388	IGCM_debug_PushStack "IGCM_sys_Tree" $@
389	if ( $DEBUG_sys ) ; then
390	echo "IGCM_sys_Tree :" $@
391	fi
392
393	\ls -lR ${@}
394
395	IGCM_debug_PopStack "IGCM_sys_Tree"
396	}
397
398	#D-#==================================================
399	#D-function IGCM_sys_Qsub
400	#D-* Purpose: Qsub new job
401	#D-* Examples:
402	#D-
403	function IGCM_sys_Qsub {
404	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
405	if ( $DEBUG_sys ) ; then
406	echo "IGCM_sys_Qsub :" $@
407	fi
408	typeset options status
409	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
410
411	/usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
412	status=$?
413
414	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
415	if [ ${status} -gt 0 ] ; then
416	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
417	IGCM_debug_Exit "IGCM_sys_Qsub"
418	else
419	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
420	fi
421	IGCM_debug_PopStack "IGCM_sys_Qsub"
422	}
423
424	#D-#==================================================
425	#D-function IGCM_sys_QsubPost
426	#D-* Purpose: Qsub new job on scalaire
427	#D-* Examples:
428	#D-
429	function IGCM_sys_QsubPost {
430	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
431	if ( $DEBUG_sys ) ; then
432	echo "IGCM_sys_QsubPost :" $@
433	fi
434	typeset options status
435	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
436
437	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
438	status=$?
439
440	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
441	if [ ${status} -gt 0 ] ; then
442	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
443	IGCM_debug_Exit "IGCM_sys_QsubPost"
444	else
445	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
446	fi
447	IGCM_debug_PopStack "IGCM_sys_QsubPost"
448	}
449
450	#D-*************************
451	#D- File transfer functions
452	#D-*************************
453	#D-
454
455	#D-#==================================================
456	#D-function IGCM_sys_RmRunDir
457	#D-* Purpose: rm tmpdir (dummy function most of the time batch
458	#D- scheduler will do the job)
459	#D-* Examples:
460	#D-
461	function IGCM_sys_RmRunDir {
462	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
463	if ( $DEBUG_sys ) ; then
464	echo "IGCM_sys_RmRunDir :" $@
465	fi
466
467	typeset status
468
469	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
470	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
471	status=$?
472
473	if [ ${status} -gt 0 ] ; then
474	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
475	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
476	IGCM_debug_Exit "IGCM_sys_RmRunDir"
477	else
478	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
479	fi
480	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
481	}
482
483	#D-#==================================================
484	#D-function IGCM_sys_Put_Dir
485	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
486	#D-* Examples:
487	#D-
488	function IGCM_sys_Put_Dir {
489	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
490	if ( $DEBUG_sys ) ; then
491	echo "IGCM_sys_Put_Dir :" $@
492	fi
493	if [ $DRYRUN = 0 ]; then
494	if [ ! -d ${1} ] ; then
495	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
496	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
497	return
498	fi
499
500	typeset status
501
502	# Only if we use rsync
503	#IGCM_sys_TestDirArchive $( dirname $2 )
504	#
505	#USUAL WAY
506	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
507	status=$?
508
509	if [ ${status} -gt 0 ] ; then
510	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
511	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
512	IGCM_debug_Exit "IGCM_sys_Put_Dir"
513	else
514	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
515	fi
516	fi
517	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
518	}
519
520	#D-#==================================================
521	#D-function IGCM_sys_Get_Dir
522	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
523	#D-* Examples:
524	#D-
525	function IGCM_sys_Get_Dir {
526	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
527	if ( $DEBUG_sys ) ; then
528	echo "IGCM_sys_Get_Dir :" $@
529	fi
530	if [ $DRYRUN = 0 ]; then
531	typeset NB_ESSAI DELAI status i
532	# number of tentative
533	NB_ESSAI=3
534	# time delay between tentative
535	DELAI=2
536
537	#
538	# USUAL WAY
539	# add here command to demigrate all offline files to reduce time of this command
540	# actually on Jean Zay files are not archive on bands
541
542	i=0
543	while [ $i -lt $NB_ESSAI ] ; do
544	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
545	status=$?
546	if [ ${status} -gt 0 ] ; then
547	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
548	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
549	sleep $DELAI
550	else
551	break
552	fi
553	(( i = i + 1 ))
554	done
555
556	if [ ${status} -gt 0 ] ; then
557	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
558	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
559	IGCM_debug_Exit "IGCM_sys_Get_Dir"
560	else
561	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
562	fi
563	fi
564	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
565	}
566
567	#D-#==================================================
568	#D-function IGCM_sys_Put_Rest
569	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
570	#D- File and target directory must exist.
571	#D-* Examples:
572	#D-
573	function IGCM_sys_Put_Rest {
574	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
575	if ( $DEBUG_sys ) ; then
576	echo "IGCM_sys_Put_Rest :" $@
577	fi
578	if [ $DRYRUN = 0 ]; then
579	if [ ! -f ${1} ] ; then
580	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
581	IGCM_debug_Exit "IGCM_sys_Put_Rest"
582	fi
583
584	typeset status
585	#
586	# USUAL WAY
587	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
588	status=$?
589
590	# #RSYNC WITH NETWORK SSH CALL
591	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
592	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593
594	# #RSYNC WITH NFS USE
595	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
596	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
597
598	# status=$?
599	# IGCM_sys_Rsync_out $status
600
601	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
602	# (( status=status+$? ))
603
604	if [ ${status} -gt 0 ] ; then
605	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
606	[ -f ${1} ] && ls -l ${1}
607	[ -f ${2} ] && ls -l ${2}
608	[ -f ${2}/${1} ] && ls -l ${2}/${1}
609	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
610	IGCM_debug_Exit "IGCM_sys_Put_Rest"
611	else
612
613	if [ X${JobType} = XRUN ] ; then
614	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
615	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
616	fi
617
618	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
619	fi
620	fi
621	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
622	}
623
624	#D-#==================================================
625	#D-function IGCM_sys_Put_Out
626	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
627	#D-* Examples:
628	#D-
629	function IGCM_sys_Put_Out {
630	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
631	if ( $DEBUG_sys ) ; then
632	echo "IGCM_sys_Put_Out :" $@
633	fi
634
635	typeset NB_ESSAI DELAI status i exist skip
636	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
637
638	# number of tentative
639	NB_ESSAI=3
640	# time delay between tentative
641	DELAI=2
642
643	if [ $DRYRUN = 0 ]; then
644	if [ ! -f ${1} ] ; then
645	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
646	IGCM_debug_PopStack "IGCM_sys_Put_Out"
647	return 1
648	fi
649	#
650	IGCM_sys_MkdirArchive $( dirname $2 )
651	#
652	exist=false
653	skip=false
654	if [ -f $2 ] ; then
655	IGCM_debug_Print 1 "$2 already exist"
656	# add here command to demigrate all offline files to reduce time of this command
657	# actually on Jean Zay files are not archive on bands
658
659	exist=true
660	if [ "X$( diff $1 $2 )" = X ] ; then
661	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
662	skip=true
663	else
664	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
665	skip=false
666	fi
667	fi
668	#
669	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
670	IGCM_sys_Chmod u+w $2
671	fi
672
673	if [ X${skip} = Xfalse ] ; then
674	i=0
675	while [ $i -lt $NB_ESSAI ] ; do
676	# Identify file system
677	fileDeviceNumberInHex=$( stat -c %d $1 )
678	status=$?
679	if [ ${status} -gt 0 ] ; then
680	IGCM_debug_Exit "IGCM_sys_Put_Out"
681	fi
682	# Identify file system
683	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
684	status=$?
685	if [ ${status} -gt 0 ] ; then
686	IGCM_debug_Exit "IGCM_sys_Put_Out"
687	fi
688
689	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
690	# They are not on the same device. USUAL WAY
691	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
692	status=$?
693	else
694	# They are on the same device. NOT SO USUAL WAY
695	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
696	status=$?
697	fi
698	if [ ${status} -gt 0 ]; then
699	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
700	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
701	[ -f ${1} ] && ls -l ${1}
702	[ -f ${2} ] && ls -l ${2}
703	[ -f ${2}/${1} ] && ls -l ${2}/${1}
704	sleep $DELAI
705	else
706	break
707	fi
708	(( i = i + 1 ))
709	done
710	fi
711
712	# #RSYNC WITH NETWORK SSH CALL
713	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
714	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
715
716	# #RSYNC WITH NFS USE
717	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
718	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
719
720	# status=$?
721	# IGCM_sys_Rsync_out $status
722
723	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
724	# (( status=status+$? ))
725
726	if [ ${status} -gt 0 ] ; then
727	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
728	[ -f ${1} ] && ls -l ${1}
729	[ -f ${2} ] && ls -l ${2}
730	[ -f ${2}/${1} ] && ls -l ${2}/${1}
731	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
732	IGCM_debug_Exit "IGCM_sys_Put_Out"
733	else
734
735	if [ X${JobType} = XRUN ] ; then
736	if [ X${3} = X ] ; then
737	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
738	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
739	fi
740	fi
741
742	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
743	fi
744	fi
745	IGCM_debug_PopStack "IGCM_sys_Put_Out"
746	return 0
747	}
748
749	#D-#==================================================
750	#D-function IGCM_sys_Get
751	#D-* Purpose: Get a file from ${ARCHIVE}
752	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
753	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
754	function IGCM_sys_Get {
755	IGCM_debug_PushStack "IGCM_sys_Get" $@
756
757	typeset DEST dm_liste target file_work
758	typeset NB_ESSAI DELAI status i
759
760	if ( $DEBUG_sys ) ; then
761	echo "IGCM_sys_Get :" $@
762	fi
763
764	# number of tentative
765	NB_ESSAI=3
766	# time delay between tentative
767	DELAI=2
768
769	if [ $DRYRUN -le 2 ]; then
770	if [ X${1} = X'/l' ] ; then
771	eval set +A dm_liste \${${2}}
772	else
773	eval set +A dm_liste ${1}
774	fi
775	eval DEST=\${${#}}
776	# add here command to demigrate all offline files to reduce time of this command
777	# actually on Jean Zay files are not archive on bands
778
779	status=$?
780	if [ ${status} -gt 0 ] ; then
781	echo "WARNING IGCM_sys_Get : error code ${status}"
782	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
783	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
784	fi
785
786	#if [ ${status} -gt 0 ] ; then
787	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
788	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
789	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
790	# sleep 30
791	# echo "We try another time"
792	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
793	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
794	# status=$?
795	# if [ ${status} -gt 0 ] ; then
796	# echo "ERROR IGCM_sys_Get : again demigration error :"
797	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
798	# IGCM_debug_Exit "IGCM_sys_Get"
799	# fi
800	# else
801	# echo "ERROR IGCM_sys_Get : demigration error :"
802	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
803	# IGCM_debug_Exit "IGCM_sys_Get"
804	# fi
805	#fi
806
807	# #RSYNC WITH NETWORK SSH CALL
808	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
809	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
810
811	# #RSYNC WITH NFS USE
812	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
813	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
814
815	# status=$?
816	# IGCM_sys_Rsync_out $status
817
818	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
819	# (( status=status+$? ))
820
821	#USUAL WAY
822	if [ X${1} = X'/l' ] ; then
823	for target in ${dm_liste[*]} ; do
824	local_file=$( basename ${target} )
825	# test if the target file is present before the loop
826	IGCM_sys_TestFileArchive ${target}
827	status=$?
828	if [ ${status} -gt 0 ] ; then
829	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
830	IGCM_debug_Exit "IGCM_sys_Get"
831	else
832	i=0
833	while [ $i -lt $NB_ESSAI ] ; do
834	#if [ X${DoLink} = Xtrue ] ; then
835	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
836	# status=$?
837	# else
838	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
839	# status=$?
840	#fi
841	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
842	status=$?
843	if [ ${status} -gt 0 ]; then
844	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
845	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
846	sleep $DELAI
847	else
848	break
849	fi
850	(( i = i + 1 ))
851	done
852	if [ ${status} -gt 0 ] ; then
853	echo "IGCM_sys_Get : error"
854	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
855	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
856	IGCM_debug_Exit "IGCM_sys_Get"
857	else
858	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
859	fi
860	fi
861	done
862	else
863	i=0
864	while [ $i -lt $NB_ESSAI ] ; do
865	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
866	status=$?
867	if [ ${status} -gt 0 ]; then
868	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
869	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
870	sleep $DELAI
871	else
872	break
873	fi
874	(( i = i + 1 ))
875	done
876	if [ ${status} -gt 0 ] ; then
877	echo "IGCM_sys_Get : error"
878	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
879	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
880	IGCM_debug_Exit "IGCM_sys_Get"
881	else
882	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
883	fi
884	fi
885	fi
886	IGCM_debug_PopStack "IGCM_sys_Get"
887	}
888
889	#D-#==================================================
890	#D-function IGCM_sys_GetDate_Monitoring
891	#D-* Purpose: get the last year for which the monitoring has been computed
892	#D-* Examples:
893	#D-
894	function IGCM_sys_GetDate_Monitoring {
895	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
896	if ( $DEBUG_sys ) ; then
897	echo "IGCM_sys_GetDate_Monitoring :" $@
898	fi
899
900	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
901
902	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
903	}
904
905	#D-#==================================================
906	#D-function IGCM_sys_Dods_Rm
907	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
908	#D-* Examples:
909	#D-
910	function IGCM_sys_Dods_Rm {
911	if ( $DEBUG_sys ) ; then
912	echo "IGCM_sys_Dods_Rm :" $@
913	fi
914	typeset status
915	if [ $DRYRUN = 0 ]; then
916
917	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
918	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
919	# echo "Nothing has been done."
920	# return
921	# fi
922
923	if [ "$#" -eq 1 ]; then
924	# $(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
925	status=$?
926	else
927	# $(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
928	status=$?
929	fi
930
931	# if [ ${status} -gt 0 ] ; then
932	# echo "IGCM_sys_Dods_Rm : error."
933	# cat out_dods_rm
934	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
935	# else
936	# rm out_dods_rm
937	# fi
938
939	fi
940	return $status
941	}
942
943	#D-#==================================================
944	#D-function IGCM_sys_Dods_Cp
945	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
946	#D-* Examples:
947	#D-
948	function IGCM_sys_Dods_Cp {
949	if ( $DEBUG_sys ) ; then
950	echo "IGCM_sys_Dods_Cp :" $@
951	fi
952	typeset status
953	if [ $DRYRUN = 0 ]; then
954
955	# if [ ! -d ${R_SAVE}/${1} ] ; then
956	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
957	# echo "Nothing has been done."
958	# return
959	# fi
960
961	# $(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
962
963	status=$?
964
965	# if [ ${status} -gt 0 ] ; then
966	# echo "IGCM_sys_Dods_Cp : error."
967	# cat out_dods_cp
968	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
969	# else
970	# rm out_dods_cp
971	# fi
972
973	fi
974	return $status
975	}
976
977	#D-#==================================================
978	#D-function IGCM_sys_Put_Dods
979	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
980	#D-* Examples:
981	#D-
982	function IGCM_sys_Put_Dods {
983	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
984	if ( $DEBUG_sys ) ; then
985	echo "IGCM_sys_Put_Dods :" $@
986	fi
987	typeset status
988	if [ $DRYRUN = 0 ]; then
989	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
990	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
991	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
992	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
993	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
994	return
995	fi
996
997	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
998	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
999	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1000	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1001	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1002	return
1003	fi
1004	#
1005	if [ -d ${R_SAVE}/${1} ] ; then
1006	cd ${R_SAVE}
1007	elif [ -d ${R_FIGR}/${1} ] ; then
1008	cd ${R_FIGR}
1009	fi
1010
1011	IGCM_sys_Dods_Rm ${1}
1012	IGCM_sys_Dods_Cp ${1}
1013	status=0
1014
1015	if [ ${status} -gt 0 ] ; then
1016	echo "IGCM_sys_Put_Dods : error."
1017	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1018	fi
1019	fi
1020	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1021	}
1022
1023	##############################################################
1024	# REBUILD OPERATOR
1025
1026	#D-#==================================================
1027	#D-function IGCM_sys_sync
1028	#D-* Purpose: flush buffer on disk
1029	#D-* Examples:
1030	#D-
1031	function IGCM_sys_sync {
1032	IGCM_debug_PushStack "IGCM_sys_sync" $@
1033	if ( $DEBUG_sys ) ; then
1034	echo "IGCM_sys_sync :" $@
1035	fi
1036
1037	/bin/sync
1038
1039	IGCM_debug_PopStack "IGCM_sys_sync"
1040	}
1041
1042	############################################################
1043	# Activate Running Environnment Variables
1044
1045	#D-#==================================================
1046	#D-function IGCM_sys_activ_variables
1047	#D-* Purpose: set environement variables prior to execution
1048	#D-* Examples:
1049	#D-
1050	function IGCM_sys_activ_variables {
1051	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1052	if ( $DEBUG_sys ) ; then
1053	echo "IGCM_sys_activ_variables"
1054	fi
1055
1056	# --------------------------------------------------------------------
1057	#D- MODULE specifications
1058	# --------------------------------------------------------------------
1059	# Source the file EnvFile containing all module load needed to run the model.
1060	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1061	# it will be used. Otherwise a default file will be used.
1062	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1063	# EnvFile is set in config.card and the file exists
1064	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1065	EnvFile=${config_UserChoices_EnvFile}
1066	else
1067	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1068	module purge > /dev/null 2>&1
1069	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1070	fi
1071	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1072	IGCM_debug_Print 1 ${EnvFile}
1073	. ${EnvFile}
1074
1075	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1076	module list
1077
1078	# --------------------------------------------------------------------
1079	#D- MPI specifications
1080	# --------------------------------------------------------------------
1081
1082	# --------------------------------------------------------------------
1083	#D- Other specifications
1084	# --------------------------------------------------------------------
1085
1086	ulimit -s unlimited
1087	if [ ${executionType} -eq 2 ] ; then
1088	export SLURM_HOSTFILE=./hostlist
1089	fi
1090
1091
1092	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1093	}
1094
1095	############################################################
1096	# Desactivate Running Environnment Variables
1097
1098	#D-#==================================================
1099	#D-function IGCM_sys_desactiv_variables
1100	#D-* Purpose: unset environement variables after execution
1101	#D-* Examples:
1102	#D-
1103	function IGCM_sys_desactiv_variables {
1104	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1105	if ( $DEBUG_sys ) ; then
1106	echo "IGCM_sys_desactiv_variables"
1107	fi
1108	# --------------------------------------------------------------------
1109	#D- MPI specifications
1110	# --------------------------------------------------------------------
1111	if [ ${executionType} -eq 2 ] ; then
1112	unset SLURM_HOSTFILE
1113	fi
1114	# --------------------------------------------------------------------
1115	#D- Other specifications
1116	# --------------------------------------------------------------------
1117
1118	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1119	}
1120
1121	############################################################
1122	# Update job headers to be used by the scheduler
1123
1124	#D-#==================================================
1125	#D-function IGCM_sys_updateHeaders
1126	#D-* Purpose: Update job headers to be used by the scheduler
1127	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1128	#D-
1129	function IGCM_sys_updateHeaders {
1130	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1131	if ( $DEBUG_sys ) ; then
1132	echo "IGCM_sys_updateHeaders"
1133	fi
1134	typeset file
1135	file=$1
1136
1137	if [ ${executionType} -eq 1 ] ; then
1138	# MPMD + MPI
1139	sed -e "/::openMPthreads::/d" \
1140	-e "s/::JobNumProcTot::/${coreNumber}/" \
1141	${file} > ${file}.tmp
1142
1143	elif [ ${executionType} -eq 2 ] ; then
1144	# MPMD + MPI + OMP
1145	(( nodeNumber = coreNumber / NB_CORE_PER_NODE ))
1146	[ $(( ${coreNumber} % ${NB_CORE_PER_NODE} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
1147	sed -e "/::openMPthreads::/d" \
1148	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1149	-e "/ntasks/i\#SBATCH --nodes=${nodeNumber}"\
1150	-e "/ntasks/i\#SBATCH --exclusive" \
1151	${file} > ${file}.tmp
1152	elif [ ${executionType} -eq 3 ] ; then
1153	# SPMD + MPI/OMP
1154	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1155	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1156	${file} > ${file}.tmp
1157
1158	elif [ ${executionType} -eq 4 ] ; then
1159	# SPMD + MPI only
1160	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1161	-e "/::openMPthreads::/d" \
1162	${file} > ${file}.tmp
1163
1164	elif [ ${executionType} -eq 5 ] ; then
1165	# SPMD + OMP only
1166	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1167	-e "/::JobNumProcTot::/d" \
1168	${file} > ${file}.tmp
1169
1170	elif [ ${executionType} -eq 6 ] ; then
1171	# SEQUENTIAL THEN
1172	sed -e "s/::JobNumProcTot::/1/" \
1173	-e "/::openMPthreads::/d" \
1174	${file} > ${file}.tmp
1175
1176	fi
1177
1178	IGCM_sys_Mv ${file}.tmp ${file}
1179
1180	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1181	}
1182
1183	############################################################
1184	# Build MPI/OMP scripts run file (dummy function)
1185
1186	#D-#==================================================
1187	#D-function IGCM_sys_build_run_file
1188	#D-* Purpose: build run file (deprecated)
1189	#D-* Examples:
1190	#D-
1191	function IGCM_sys_build_run_file {
1192	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1193	}
1194
1195	############################################################
1196	# Build MPI/OMP scripts
1197
1198	#D-#==================================================
1199	#D-function IGCM_sys_build_execution_scripts
1200	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1201	#D-* Examples:
1202	#D-
1203	function IGCM_sys_build_execution_scripts
1204	{
1205	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1206	if ( $DEBUG_sys ) ; then
1207	echo "IGCM_sys_build_execution_scripts " $@
1208	fi
1209	IGCM_debug_Print 1 "executionType= ${executionType}"
1210	EXECUTION=${HOST_MPIRUN_COMMAND}
1211
1212	if ( ${OK_PARA_MPMD} ) ; then
1213
1214	# MPMD mode
1215	# 1 MPI only : executionType=1
1216	# 2 MPI/OpenMP : executionType=2
1217
1218	if [ -f run_file ] ; then
1219	IGCM_sys_Rm -f run_file
1220	fi
1221	touch run_file
1222
1223	# case 1 : Only MPI (MPMD)
1224	if ( ! ${OK_PARA_OMP} ) ; then
1225	# Build run_file
1226	current_core=0
1227	# First loop on the components for the coupler ie oasis (only if oasis3)
1228	# the coupler ie oasis3 must be the first one
1229	for comp in ${config_ListOfComponents[*]} ; do
1230
1231	eval ExeNameIn=\${config_Executable_${comp}[0]}
1232	eval ExeNameOut=\${config_Executable_${comp}[1]}
1233
1234	# Only if the component has an executable
1235	if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1236
1237	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1238	(( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
1239	echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
1240	(( current_core = ${end_core} + 1 ))
1241	fi
1242	done
1243
1244	EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
1245
1246	IGCM_sys_Chmod u+x run_file
1247	if ( $DEBUG_sys ) ; then
1248	echo "run_file contains : "
1249	cat run_file
1250	fi
1251
1252	else
1253
1254	# 2 MPI/OpenMP : executionType=2
1255
1256	# MPI-OpenMP (MPMD)
1257	# export SLURM_HOSTFILE=./hostlist
1258	# srun --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file
1259	# example of hostlist file :
1260	# r3i3n33
1261	# r3i3n33
1262	# ...
1263	# example of run_file :
1264	# 0 taskset -c 0-4 ./script_lmdz.x.ksh
1265	# 1 taskset -c 5-9 ./script_lmdz.x.ksh
1266	#...
1267	# 32 taskset -c 0 ./script_opa.xx.ksh
1268	# 33 taskset -c 1 ./script_opa.xx.ksh
1269	#...
1270	# 62 taskset -c 30 ./script_xios.x.ksh
1271
1272	# Hosts treatment
1273	IFS=$'\n'; set -f
1274	listnodes=($(< <( scontrol show hostnames $SLURM_JOB_NODELIST )))
1275	unset IFS; set +f
1276	rm -f hostlist
1277
1278	# Loop on the components to build run_file and script_exec files
1279	rank=0
1280	current_core=0
1281
1282	for comp in ${config_ListOfComponents[*]} ; do
1283
1284	eval ExeNameIn=\${config_Executable_${comp}[0]}
1285	eval ExeNameOut=\${config_Executable_${comp}[1]}
1286
1287	# Not possible if oasis has an executable (i.e old version of oasis3)
1288	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1289	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1290	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1291	IGCM_debug_Verif_Exit
1292	fi
1293
1294	# Only if we really have an executable for the component :
1295	if [ "X${ExeNameOut}" != X\"\" ] ; then
1296
1297	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1298	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1299	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1300
1301
1302	# Build script files
1303
1304	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1305	echo "" >> script_${ExeNameOut}.ksh
1306	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1307	# Check if the number of threads is correct
1308	case ${comp_proc_omp_loc} in
1309	2\|4\|5\|10\|20)
1310	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1311	;;
1312	*)
1313	IGCM_debug_Exit "ERROR with OMP parameters !"
1314	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1315	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1316	IGCM_debug_Verif_Exit
1317	;;
1318	esac
1319	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1320	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
1321	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1322
1323	fi
1324
1325	# to have out/err per process on different files
1326	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1327
1328	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1329
1330	# Complete run_file
1331	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1332	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1333	else
1334	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1335	fi
1336
1337	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1338	(( index_host = current_core / NB_CORE_PER_NODE ))
1339	host_value=${listnodes[${index_host}]}
1340	(( slot = current_core % NB_CORE_PER_NODE ))
1341	echo "$host_value" >> hostlist
1342	echo "$rank taskset -c $slot-$((slot+comp_proc_omp_loc-1)) ./script_${ExeNameOut}.ksh" >> run_file
1343	(( rank = rank + 1 ))
1344	(( current_core = current_core + offset_comp_proc_loc ))
1345	done
1346	fi
1347	done
1348
1349	## variable added to stop after 60s instead of 600s by default.
1350	## This is used when no error comes from executables and when something stopped an executable without notice.
1351	export SLURM_WAIT=60
1352
1353	EXECUTION="${HOST_MPIRUN_COMMAND} --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file"
1354
1355	IGCM_sys_Chmod u+x run_file
1356	if ( $DEBUG_sys ) ; then
1357	echo "run_file contains : "
1358	cat run_file
1359	fi
1360
1361	fi # if ${OK_PARA_MPMD}
1362
1363	else
1364	# Only one executable (SPMD mode): executionType=3, 4, 5 and 6
1365
1366	for comp in ${config_ListOfComponents[*]} ; do
1367
1368	# Only if we really have an executable for the component :
1369	eval ExeNameOut=\${config_Executable_${comp}[1]}
1370	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1371
1372	# Build script files
1373
1374	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1375	echo "" >> script_${ExeNameOut}.ksh
1376	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1377
1378	if ( ${OK_PARA_OMP} ) ; then
1379	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1380	# Check if the number of threads is correct
1381	case ${comp_proc_omp_loc} in
1382	2\|4\|5\|10\|20)
1383	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1384	;;
1385	*)
1386	IGCM_debug_Exit "ERROR with OMP parameters !"
1387	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1388	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1389	IGCM_debug_Verif_Exit
1390	;;
1391	esac
1392	echo "" >> script_${ExeNameOut}.ksh
1393	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1394	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
1395	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1396	fi
1397
1398	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1399
1400	# To have out/err per process on different files
1401	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1402	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1403
1404	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1405	cat script_${ExeNameOut}.ksh
1406
1407	fi
1408
1409	done
1410
1411	fi # ${OK_PARA_MPMD}
1412
1413	IGCM_debug_Print 1 "sys Irene : execution command is "
1414	IGCM_debug_Print 1 "$EXECUTION"
1415
1416	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1417	}
1418
1419	#D-#==================================================
1420	#D-function IGCM_sys_check_path
1421	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1422	#D-* do not point to an important use directory. Stop immediately in that case.
1423	#D-* Examples:
1424	#D-
1425	function IGCM_sys_check_path {
1426	IGCM_debug_PushStack "IGCM_sys_check_path"
1427	if ( $DEBUG_sys ) ; then
1428	echo "IGCM_sys_check_path"
1429	fi
1430
1431	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1432	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1433	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1434	IGCM_debug_Exit "This will stop the job"
1435	elif ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X/gpfswork/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/gpfsscratch/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/gpfsstore/rech/${DataProject}/${LOGIN} ] ) ; then
1436	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1437	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1438	IGCM_debug_Exit "This will stop the job"
1439	fi
1440	IGCM_debug_PopStack "IGCM_sys_check_path"
1441	}
1442
1443	#D-#==================================================
1444	#D-function IGCM_sys_check_quota
1445	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1446	#D-* Examples:
1447	#D-
1448	function IGCM_sys_check_quota {
1449	IGCM_debug_PushStack "IGCM_sys_check_quota"
1450	echo "IGCM_sys_check_quota is desactivated on JeanZay because there is actually no way to check scratch quota on this computing center (2019.11.19)"
1451	# if ( $DEBUG_sys ) ; then
1452	# echo "IGCM_sys_check_quota"
1453	# fi
1454	# # Limit of quota (in %)
1455	# limit_quota=90
1456	#
1457	# # Check of the volume
1458	# volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1459	# volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1460	#
1461	# if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1462	#
1463	# unit_avail=${volume_avail: -1}
1464	# unit_quota=${volume_quota: -1}
1465	#
1466	# if [ "${unit_quota}" = "*" ] ; then
1467	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1468	# IGCM_debug_Print 1 "More than 100% of your quota is used"
1469	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1470	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1471	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1472	# IGCM_debug_Verif_Exit
1473	# fi
1474	#
1475	# temp_avail=${volume_avail%%${volume_avail: -1}*}
1476	# temp_quota=${volume_quota%%${volume_quota: -1}*}
1477	#
1478	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
1479	#
1480	# # Convertion
1481	# if [ ${volume_avail: -1} = "T" ] ; then
1482	# (( temp_avail = temp_avail * 1000000000000 ))
1483	# elif [ ${volume_avail: -1} = "G" ] ; then
1484	# (( temp_avail = temp_avail * 1000000000 ))
1485	# elif [ ${volume_avail: -1} = "M" ] ; then
1486	# (( temp_avail = temp_avail * 1000000 ))
1487	# elif [ ${volume_avail: -1} = "k" ] ; then
1488	# (( temp_avail = temp_avail * 1000 ))
1489	# else
1490	# (( temp_avail = volume_avail ))
1491	# fi
1492	# if [ ${volume_quota: -1} = "T" ] ; then
1493	# (( temp_quota = temp_quota * 1000000000000 ))
1494	# elif [ ${volume_quota: -1} = "G" ] ; then
1495	# (( temp_quota = temp_quota * 1000000000 ))
1496	# elif [ ${volume_quota: -1} = "M" ] ; then
1497	# (( temp_quota = temp_quota * 1000000 ))
1498	# elif [ ${volume_quota: -1} = "k" ] ; then
1499	# (( temp_quota = temp_quota * 1000 ))
1500	# else
1501	# (( temp_quota = volume_quota ))
1502	# fi
1503	# fi
1504	#
1505	# quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1506	## echo "volume ratio is " $quota_volume
1507	#
1508	# if [ ${quota_volume} -ge ${limit_quota} ] ; then
1509	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1510	# IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1511	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1512	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1513	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1514	# IGCM_debug_Verif_Exit
1515	# fi
1516	#
1517	# fi
1518	#
1519	## Check of the number of inodes
1520	#
1521	# inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1522	# inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1523	#
1524	# if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1525	#
1526	# unit_avail=${inode_avail: -1}
1527	# unit_quota=${inode_quota: -1}
1528	#
1529	# if [ "${unit_quota}" = "*" ] ; then
1530	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1531	# IGCM_debug_Print 1 "More than 100% of your quota is used"
1532	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1533	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1534	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1535	# IGCM_debug_Verif_Exit
1536	# fi
1537	#
1538	# temp_avail=${inode_avail%%${inode_avail: -1}*}
1539	# temp_quota=${inode_quota%%${inode_quota: -1}*}
1540	#
1541	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
1542	#
1543	# # Convertion
1544	# if [ ${inode_avail: -1} = "T" ] ; then
1545	# (( temp_avail = temp_avail * 1000000000000 ))
1546	# elif [ ${inode_avail: -1} = "G" ] ; then
1547	# (( temp_avail = temp_avail * 1000000000 ))
1548	# elif [ ${inode_avail: -1} = "M" ] ; then
1549	# (( temp_avail = temp_avail * 1000000 ))
1550	# elif [ ${inode_avail: -1} = "k" ] ; then
1551	# (( temp_avail = temp_avail * 1000 ))
1552	# else
1553	# (( temp_avail = inode_avail ))
1554	# fi
1555	#
1556	# if [ ${inode_quota: -1} = "T" ] ; then
1557	# (( temp_quota = temp_quota * 1000000000000 ))
1558	# elif [ ${inode_quota: -1} = "G" ] ; then
1559	# (( temp_quota = temp_quota * 1000000000 ))
1560	# elif [ ${inode_quota: -1} = "M" ] ; then
1561	# (( temp_quota = temp_quota * 1000000 ))
1562	# elif [ ${inode_quota: -1} = "k" ] ; then
1563	# (( temp_quota = temp_quota * 1000 ))
1564	# else
1565	# (( temp_quota = inode_quota ))
1566	# fi
1567	# fi
1568	# quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1569	## echo "inode ratio is " $quota_inode
1570	#
1571	# if [ ${quota_inode} -ge ${limit_quota} ] ; then
1572	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1573	# IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1574	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1575	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1576	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1577	# IGCM_debug_Verif_Exit
1578	# fi
1579	# fi
1580	IGCM_debug_PopStack "IGCM_sys_check_quota"
1581	}
1582
1583	#D-#==================================================
1584	#D-function IGCM_sys_projectAccounting
1585	#D-* Purpose: store project accounting information in a file
1586	#D-* Examples:
1587	#D-
1588	function IGCM_sys_projectAccounting {
1589	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1590	if ( $DEBUG_sys ) ; then
1591	echo "IGCM_sys_check_quota"
1592	fi
1593	#need to implemented on Jean Zay
1594	touch $1
1595	# ssh irene191 /usr/bin/ccc_myproject > $1
1596
1597	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1598	}
1599
1600	#D-#==================================================
1601	#D-function IGCM_sys_getJobSchedulerID
1602	#D-* Purpose: Get the job ID during execution
1603	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1604	#D-
1605	function IGCM_sys_getJobSchedulerID {
1606	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1607	if ( $DEBUG_sys ) ; then
1608	echo "IGCM_sys_getJobSchedulerID"
1609	fi
1610
1611	eval ${1}=${SLURM_JOBID}
1612
1613	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1614	}
1615
1616	#D-#==================================================
1617	#D-function IGCM_sys_GetJobID
1618	#D-* Purpose: Get the job ID from the JobName
1619	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1620	#D-
1621	function IGCM_sys_GetJobID {
1622	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1623	if ( $DEBUG_sys ) ; then
1624	echo "IGCM_sys_GetJobID"
1625	fi
1626
1627	ID=$( squeue -u $user \| grep Job_${config_UserChoices_JobName} \| gawk ' { print $1}' )
1628
1629
1630	eval ${3}=${ID}
1631	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1632	}
1633
1634	#D-#==================================================
1635	#D-function IGCM_sys_CountJobInQueue
1636	#D-* Purpose: Count number of users job
1637	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1638	#D-
1639	function IGCM_sys_CountJobInQueue {
1640	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1641	if ( $DEBUG_sys ) ; then
1642	echo "IGCM_sys_CountJobInQueue"
1643	fi
1644
1645	# With -f option, the full job name is given in the last column
1646	NbRun=$(squeue -u $user \| \
1647	grep -v JOBID \| gawk 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1648
1649	eval ${2}=${NbRun}
1650
1651	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1652	}
1653
1654	#D-#==================================================
1655	#D-function IGCM_sys_ListJobInQueue
1656	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1657	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1658	#D-
1659	function IGCM_sys_ListJobInQueue {
1660	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1661	if ( $DEBUG_sys ) ; then
1662	echo "IGCM_sys_ListJobInQueue"
1663	fi
1664
1665	# to be implemented on Jean Zay
1666	set -A JobList $( squeue -u $user \| gawk '{print $3}' \| grep -v NAME)
1667	# # With -f option, the full job name is given in the last column
1668	# set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1669	# '( $2 == User && \
1670	# $NF != /TS/ && \
1671	# $NF !~ /PACK/ && \
1672	# $NF !~ /REBUILD/ && \
1673	# $NF !~ /pack/ ) \
1674	# { print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1675	#
1676	eval set -A ${2} ${JobList[*]}
1677
1678	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1679	}
1680
1681	#D-#==================================================
1682	#D-function IGCM_sys_atlas
1683	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1684	#D-* Examples:
1685	#D-
1686	function IGCM_sys_atlas {
1687	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1688	if ( $DEBUG_sys ) ; then
1689	echo "IGCM_sys_atlas :" $@
1690	fi
1691
1692	typeset status
1693	# To be implemented on Jean Zay
1694	#
1695	# \ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1696	# status=$?
1697	# if [ ${status} -gt 0 ] ; then
1698	# echo "IGCM_sys_atlas : error code ${status}"
1699	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1700	# \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1701	# IGCM_debug_PopStack "IGCM_sys_atlas"
1702	# return 1
1703	# else
1704	# IGCM_debug_PopStack "IGCM_sys_atlas"
1705	# return 0
1706	# fi
1707	#
1708	IGCM_debug_PopStack "IGCM_sys_atlas"
1709	}
1710
1711	#D-#==================================================
1712	#D-function IGCM_sys_rebuild_nemo
1713	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1714	#D-* Examples:
1715	#D-
1716
1717	function IGCM_sys_rebuild_nemo {
1718	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1719	if ( $DEBUG_sys ) ; then
1720	echo "IGCM_sys_rebuild_nemo :" $@
1721	fi
1722
1723	/gpfswork/rech/psl/commun/Tools/rebuild_nemo/bin/rebuild_nemo ${1} ${2}
1724
1725	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1726	}
1727

Note: See TracBrowser for help on using the repository browser.

Download in other formats: