Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1659

Last change on this file since 1659 was 1659, checked in by aclsce, 3 months ago
Merged with branches/libIGCM_PREPOST to split main Job into 3 Jobs (prerun, compute and postrun) on JeanZay? supercomputer at IDRIS. Prerun and postrun run on prepost partition to have access to STORE space.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 64.6 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Jean-Zay
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
41	typeset -r DRYRUN=0
42	typeset -r DRYRUN_DEBUG=4
43	else
44	typeset -r DRYRUN=${DRYRUN:=0}
45	typeset -r DRYRUN_DEBUG=0
46	fi
47
48	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
49	# -------------------------------------------------------------------------------------
50	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
51	# \| \| Cp/Exe/param/files \| \| \| \|
52	# \| \| Chmod Qsub \| \| \| \|
53	# -------------------------------------------------------------------------------------
54	# \| 0 \| yes \| yes \| yes \| yes \|
55	# -------------------------------------------------------------------------------------
56	# \| 1 \| yes \| yes \| yes \| no \|
57	# -------------------------------------------------------------------------------------
58	# \| 2 \| yes \| yes \| no \| no \|
59	# -------------------------------------------------------------------------------------
60	# \| 3 \| yes \| no \| no \| no \|
61	# -------------------------------------------------------------------------------------
62	# \| 4 \| yes \| yes \| creation of RUN_DIR and Job_debug \|
63	# -------------------------------------------------------------------------------------
64
65
66	#=====================================================
67	# Global Variables :
68	#=====================================================
69	# Language : "fr" or "en"
70	typeset -r MYLANG="fr"
71
72	#=====================================================
73	# Host user names project maxCpuTime
74	# $hostname ou hostname
75	typeset HOST=${HOST:=$( hostname )}
76	# $username ou whoami
77	typeset LOGIN=${LOGIN:=$( whoami )}
78	# $hostname of the MASTER job
79	typeset MASTER=jeanzay
80	# add default project set by ins_job
81	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
82	### jobWarningDelay in seconds
83	typeset jobWarningDelayMin=$(squeue -j $SLURM_JOB_ID -h --Format TimeLimit \| awk -F: '{print $1}')
84	typeset jobWarningDelay=$( echo "scale=6;${jobWarningDelayMin=}*60" \| bc )
85
86	#D-
87	#D-#==================================================
88	#D-Program used in libIGCM
89	#D-#==================================================
90
91	# Submit command
92	typeset SUBMIT=${SUBMIT:=sbatch}
93	# rsync with path
94	typeset -r RSYNC=/usr/bin/rsync
95	# RSYNC_opt args to rsync
96	typeset -r RSYNC_opt="-va"
97	# ie storage filesystem
98	typeset -r STOREHOST=${MASTER}
99	typeset -r REMOTE_RSYNC=/usr/bin/rsync
100
101	#====================================================
102	# Access to module command
103	#====================================================
104	# See ticket #348
105	# Following path should be used according to assist IDRIS
106	. /gpfslocalsup/spack_soft/environment-modules/current/init/ksh
107	# Following old path gives error messages while loading modules but it still works
108	#. /usr/share/Modules/init/ksh
109
110	#====================================================
111	# Set environment tools (ferret, nco, cdo, rebuild, ...)
112	#====================================================
113	if [ X${TaskType} = Xcomputing ] ; then
114	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
115	else
116	module purge > /dev/null 2>&1
117	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
118
119	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
120
121	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
122	export UVCDAT_ANONYMOUS_LOG=FALSE
123	fi
124
125	# FYI
126	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
127	[ ! X${TaskType} = Xchecking ] && module list
128
129	# For AddNoise (not yet AddPertu3DOCE)
130	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/AddNoise/bin
131
132	# For rebuild
133	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
134
135	#====================================================
136	# Host specific DIRECTORIES
137	#====================================================
138
139	#====================================================
140	#- MirrorlibIGCM for frontend
141	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
142
143	#====================================================
144	#- libIGCM_POST for frontend
145	typeset -r libIGCM_POST=${libIGCM}
146
147	#====================================================
148	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
149	typeset -r R_EXE="${MODIPSL}/bin"
150
151	#====================================================
152	#- SUBMIT_DIR : submission dir
153	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
154
155	#====================================================
156	#- IN
157	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
158
159	#====================================================
160	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
161	typeset RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
162
163	typeset job_type=${job_type:="prerun"}
164
165	#====================================================
166	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
167	typeset OUTCOMMAND_PATH=/tmp
168
169	#====================================================
170	#- HOST_MPIRUN_COMMAND
171	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
172
173	#====================================================
174	#- Max number of arguments passed to nco operator or demigration command
175	UNIX_MAX_LIMIT=360
176
177	#====================================================
178	#- set PackDefault to true
179	PackDefault=true
180
181	#====================================================
182	#- Default number of MPI task for IPSL coupled model
183	#- required for backward compatibility
184	#-
185	DEFAULT_NUM_PROC_OCE=5
186	DEFAULT_NUM_PROC_CPL=1
187	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
188
189	#====================================================
190	#- Number of computing cores per node
191	#-
192	NB_CORE_PER_NODE=40
193
194	#D-#==================================================
195	#D-function IGCM_sys_defineArchives
196	#D-* Purpose:
197	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
198	#D-* Define ARCHIVE : Dedicated to large files
199	#D-* Define STORAGE : Dedicated to small/medium files
200	#D-* Define R_OUT : Output tree located on ARCHIVE
201	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
202	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
203	#D-* if SpaceName=TEST everything is stored on SCRATCH
204	#D-* Examples:
205	#D-
206	function IGCM_sys_defineArchives {
207	IGCM_debug_PushStack "IGCM_sys_defineArchives"
208
209	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
210	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = XDEFAULT ]; then
211	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
212	# Default option: Set DataProject according to the project used for submission
213	DataProject=${PROJECT}
214	else
215	# Use the project set by the variable DataProject in section UserChoices in config.card
216	IGCM_debug_Print 1 "Use DataProject as set in config.card : ${config_UserChoices_DataProject}"
217	DataProject=${config_UserChoices_DataProject}
218	fi
219	IGCM_debug_Print 1 "Data project for output to be used: ${DataProject}"
220
221
222	#====================================================
223	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
224	#- DataProject = for is the dataproject for Training account at Idris
225	#- they are not store in rech but in another partition
226	if [ X${DataProject} = Xfor ]; then
227	RUN_DIR_PATH=$SCRATCH/RUN_DIR/${SLURM_JOBID}_${$}
228	else
229	RUN_DIR_PATH=${RUN_DIR_PATH:=/lustre/fsn1/projects/rech/${DataProject}/${LOGIN}/RUN_DIR/${SLURM_JOBID}_${$}}
230	fi
231
232	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
233	#====================================================
234	#- ARCHIVE (dedicated to large files)
235	ARCHIVE=${config_UserChoices_ARCHIVE}
236	else
237	if [ X${DataProject} = Xfor ]; then
238	ARCHIVE=$STORE
239	else
240	#====================================================
241	#- ARCHIVE (dedicated to large files)
242	ARCHIVE=/lustre/fsstor/projects/rech/${DataProject}/${LOGIN}
243	fi
244	fi
245
246	if [ ! X${config_UserChoices_STORAGE} = X ]; then
247	#====================================================
248	#- STORAGE (dedicated to small/medium files)
249	STORAGE=${config_UserChoices_STORAGE}
250	else
251	#====================================================
252	#- STORAGE (dedicated to small/medium files)
253	if [ X${DataProject} = Xfor ]; then
254	STORAGE=$WORK
255	else
256	STORAGE=/gpfswork/rech/${DataProject}/${LOGIN}
257	fi
258	fi
259
260	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
261	#====================================================
262	#- R_OUT
263	if [ X${DataProject} = Xfor ]; then
264	R_OUT=$SCRATCH/IGCM_OUT
265	else
266	R_OUT=/lustre/fsn1/projects/rech/${DataProject}/${LOGIN}/IGCM_OUT
267	fi
268	#====================================================
269	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
270	R_FIG=${R_OUT}
271	if [ ! X${TMPARCHIVE} = X ]; then
272	R_TMPOUT=${R_OUT}
273	fi
274	if [ ! X${OLDARCHIVE} = X ]; then
275	R_OLDOUT=${OLDARCHIVE}/IGCM_OUT
276	fi
277	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
278
279	else
280	#====================================================
281	#- R_OUT
282	R_OUT=${ARCHIVE}/IGCM_OUT
283	if [ ! X${TMPARCHIVE} = X ]; then
284	R_TMPOUT=${TMPARCHIVE}/IGCM_OUT
285	fi
286	if [ ! X${OLDARCHIVE} = X ]; then
287	R_OLDOUT=${OLDARCHIVE}/IGCM_OUT
288	fi
289	#====================================================
290	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
291	R_FIG=${STORAGE}/IGCM_OUT
292	fi
293
294	#====================================================
295	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
296	CMIP6_BUF=${STORAGE}/IGCM_OUT
297
298	#====================================================
299	#- R_BUF (ONLY FOR double copy an scratch)
300
301	if [ X${DataProject} = Xfor ]; then
302	R_BUF=$SCRATCH/IGCM_OUT
303	else
304	R_BUF=/lustre/fsn1/projects/rech/${DataProject}/${LOGIN}/IGCM_OUT
305	fi
306
307	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
308	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
309	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
310
311	IGCM_debug_PopStack "IGCM_sys_defineArchives"
312	}
313
314	#D-#==================================================
315	#D-function IGCM_sys_RshArchive
316	#D-* Purpose: Archive rsh command
317	#D-* Examples:
318	#D-
319	function IGCM_sys_RshArchive {
320	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
321	/bin/ksh <<-EOF
322	${@}
323	EOF
324	status=$?
325	if [ ${status} -gt 0 ] ; then
326	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
327	IGCM_debug_Exit "IGCM_sys_RshArchive"
328	fi
329	IGCM_debug_PopStack "IGCM_sys_RshArchive"
330	}
331
332	#D-#==================================================
333	#D-function IGCM_sys_RshArchive_NoError
334	#D-* Purpose: Archive rsh command, without error
335	#D-* used only in monitoring.job
336	#D-* Examples:
337	#D-
338	function IGCM_sys_RshArchive_NoError {
339	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
340	/bin/ksh <<-EOF
341	${@} 2> /dev/null
342	EOF
343	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
344	}
345
346	#D-#==================================================
347	#D-function IGCM_sys_MkdirArchive
348	#D-* Purpose: Mkdir on Archive
349	#D-* Examples:
350	#D-
351	function IGCM_sys_MkdirArchive {
352	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
353	if ( $DEBUG_sys ) ; then
354	echo "IGCM_sys_MkdirArchive :" $@
355	fi
356	#- creation de repertoire sur le serveur fichier
357	if [ ! -d ${1} ]; then
358	\mkdir -p $1
359	status=$?
360
361	if [ ${status} -gt 0 ] ; then
362	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
363	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
364	fi
365	fi
366	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
367	}
368
369	#D-#==================================================
370	#D-function IGCM_sys_TestDirArchive
371	#D-* Purpose: Test Directory that must exists on Archive
372	#D-* Examples:
373	#D-
374	function IGCM_sys_TestDirArchive {
375	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
376	if ( $DEBUG_sys ) ; then
377	echo "IGCM_sys_TestDirArchive :" $@
378	fi
379	typeset ExistFlag
380	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
381	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
382	return ${ExistFlag}
383	}
384
385	#D-#==================================================
386	#D-function IGCM_sys_IsFileArchived
387	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
388	#D-* Examples:
389	#D-
390	function IGCM_sys_IsFileArchived {
391	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
392	if ( $DEBUG_sys ) ; then
393	echo "IGCM_sys_IsFileArchived :" $@
394	fi
395	typeset IsArchivedFlag
396	# To be implemented on Jean Zay
397	# IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
398	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
399
400	return ${IsArchivedFlag}
401	}
402
403	#D-#==================================================
404	#D-function IGCM_sys_TestFileArchive
405	#D-* Purpose: Test file that must NOT EXISTS on Archive
406	#D-* Examples:
407	#D-
408	function IGCM_sys_TestFileArchive {
409	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
410	typeset ExistFlag
411	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
412	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
413
414	return ${ExistFlag}
415	}
416
417	#D-#==================================================
418	#D-function IGCM_sys_CountFileArchive
419	#D-* Purpose: Count files on Archive filesystem
420	#D-* Examples:
421	#D-
422	function IGCM_sys_CountFileArchive {
423	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
424	ls ${@} 2>/dev/null \| wc -l
425	if [ $? -gt 0 ] ; then
426	echo "IGCM_sys_CountFileArchive : erreur."
427	fi
428	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
429	}
430
431	#D-#==================================================
432	#D-function IGCM_sys_Tree
433	#D-* Purpose: Tree directories with files on ${ARCHIVE}
434	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
435	#D-
436	function IGCM_sys_Tree {
437	IGCM_debug_PushStack "IGCM_sys_Tree" $@
438	if ( $DEBUG_sys ) ; then
439	echo "IGCM_sys_Tree :" $@
440	fi
441
442	\ls -lR ${@}
443
444	IGCM_debug_PopStack "IGCM_sys_Tree"
445	}
446
447	#D-#==================================================
448	#D-function IGCM_sys_Qsub
449	#D-* Purpose: Qsub new job
450	#D-* Examples:
451	#D-
452	function IGCM_sys_Qsub {
453	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
454	if ( $DEBUG_sys ) ; then
455	echo "IGCM_sys_Qsub :" $@
456	fi
457	typeset options status
458	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
459	export job_type="prerun"
460	unset RUN_DIR
461
462	sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
463	status=$?
464
465	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
466	if [ ${status} -gt 0 ] ; then
467	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
468	IGCM_debug_Exit "IGCM_sys_Qsub"
469	else
470	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
471	fi
472	IGCM_debug_PopStack "IGCM_sys_Qsub"
473	}
474
475	#D-#==================================================
476	#D-function IGCM_sys_Qsub_job_compute
477	#D-* Purpose: Qsub new computing job
478	#D-* Examples:
479	#D-
480	function IGCM_sys_Qsub_job_compute {
481	IGCM_debug_PushStack "IGCM_sys_Qsub_job_compute" $@
482	if ( $DEBUG_sys ) ; then
483	echo "IGCM_sys_Qsub_job_compute :" $@
484	fi
485	typeset options status
486	(( requested_time_seconds = SLURM_JOB_END_TIME - SLURM_JOB_START_TIME ))
487	(( requested_time_minute = requested_time_seconds / 60 ))
488	options="-o ${SUBMIT_DIR}/Script_Output_${config_UserChoices_JobName}.$( printf "%06d" ${CumulPeriod} ) -e ${SUBMIT_DIR}/Script_Output_${config_UserChoices_JobName}.$( printf "%06d" ${CumulPeriod} ) --open-mode=append --time=${requested_time_minute}"
489
490	export job_type="compute"
491	export RUN_DIR=${RUN_DIR}
492	export EXPERIMENT=${SUBMIT_DIR}
493	export CumulPeriod=${CumulPeriod}
494	export FirstInitialize=${FirstInitialize}
495	export PREFIX=${PREFIX}
496	unset SLURM_HOSTFILE
497
498	sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
499	status=$?
500
501	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
502	if [ ${status} -gt 0 ] ; then
503	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
504	IGCM_debug_Exit "IGCM_sys_Qsub"
505	else
506	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
507	fi
508	IGCM_debug_PopStack "IGCM_sys_Qsub_job_compute"
509	}
510
511	#D-#==================================================
512	#D-function IGCM_sys_Qsub_postrun
513	#D-* Purpose: Qsub new job
514	#D-* Examples:
515	#D-
516	function IGCM_sys_Qsub_job_postrun {
517	IGCM_debug_PushStack "IGCM_sys_Qsub_job_postrun" $@
518	if ( $DEBUG_sys ) ; then
519	echo "IGCM_sys_Qsub_job_postrun :" $@
520	fi
521	typeset options status
522	options="-o ${EXPERIMENT}/Script_Output_${JobName}.$( printf "%06d" ${CumulPeriod} ) -e ${EXPERIMENT}/Script_Output_${JobName}.$( printf "%06d" ${CumulPeriod} ) --open-mode=append"
523
524	export job_type="postrun"
525	export RUN_DIR=${RUN_DIR}
526	export ExecutionFail=${ExecutionFail}
527	export FirstInitialize=${FirstInitialize}
528	export executionType=${executionType}
529	unset SLURM_HOSTFILE
530
531	sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
532	status=$?
533
534	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
535	if [ ${status} -gt 0 ] ; then
536	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
537	IGCM_debug_Exit "IGCM_sys_Qsub"
538	else
539	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
540	fi
541	IGCM_debug_PopStack "IGCM_sys_Qsub_job_postrun"
542	}
543
544	#D-#==================================================
545	#D-function IGCM_sys_QsubPost
546	#D-* Purpose: Qsub new job on scalaire
547	#D-* Examples:
548	#D-
549	function IGCM_sys_QsubPost {
550	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
551
552	BAK_DIR=$PWD
553	cd ${SUBMIT_DIR}
554
555	if ( $DEBUG_sys ) ; then
556	echo "IGCM_sys_QsubPost :" $@
557	fi
558	typeset options status
559	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
560
561	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
562	status=$?
563
564	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
565	if [ ${status} -gt 0 ] ; then
566	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
567	IGCM_debug_Exit "IGCM_sys_QsubPost"
568	else
569	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
570	fi
571
572	cd ${BAK_DIR}
573
574	IGCM_debug_PopStack "IGCM_sys_QsubPost"
575	}
576
577	#D-*************************
578	#D- File transfer functions
579	#D-*************************
580	#D-
581
582	#D-#==================================================
583	#D-function IGCM_sys_RmRunDir
584	#D-* Purpose: rm tmpdir (dummy function most of the time batch
585	#D- scheduler will do the job)
586	#D-* Examples:
587	#D-
588	function IGCM_sys_RmRunDir {
589	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
590	if ( $DEBUG_sys ) ; then
591	echo "IGCM_sys_RmRunDir :" $@
592	fi
593
594	typeset status
595
596	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
597	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
598	status=$?
599
600	if [ ${status} -gt 0 ] ; then
601	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
602	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
603	IGCM_debug_Exit "IGCM_sys_RmRunDir"
604	else
605	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
606	fi
607	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
608	}
609
610	#D-#==================================================
611	#D-function IGCM_sys_Put_Dir
612	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
613	#D-* Examples:
614	#D-
615	function IGCM_sys_Put_Dir {
616	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
617	if ( $DEBUG_sys ) ; then
618	echo "IGCM_sys_Put_Dir :" $@
619	fi
620	if [ $DRYRUN = 0 ]; then
621	if [ ! -d ${1} ] ; then
622	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
623	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
624	return
625	fi
626
627	typeset status
628
629	# Only if we use rsync
630	#IGCM_sys_TestDirArchive $( dirname $2 )
631	#
632	#USUAL WAY
633	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
634	status=$?
635
636	if [ ${status} -gt 0 ] ; then
637	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
638	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
639	IGCM_debug_Exit "IGCM_sys_Put_Dir"
640	else
641	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
642	fi
643	fi
644	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
645	}
646
647	#D-#==================================================
648	#D-function IGCM_sys_Get_Dir
649	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
650	#D-* Examples:
651	#D-
652	function IGCM_sys_Get_Dir {
653	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
654	if ( $DEBUG_sys ) ; then
655	echo "IGCM_sys_Get_Dir :" $@
656	fi
657	if [ $DRYRUN = 0 ]; then
658	typeset NB_ESSAI DELAI status i
659	# number of tentative
660	NB_ESSAI=3
661	# time delay between tentative
662	DELAI=2
663
664	#
665	# USUAL WAY
666	# add here command to demigrate all offline files to reduce time of this command
667	# actually on Jean Zay files are not archive on bands
668
669	i=0
670	while [ $i -lt $NB_ESSAI ] ; do
671	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
672	status=$?
673	if [ ${status} -gt 0 ] ; then
674	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
675	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
676	sleep $DELAI
677	else
678	break
679	fi
680	(( i = i + 1 ))
681	done
682
683	if [ ${status} -gt 0 ] ; then
684	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
685	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
686	IGCM_debug_Exit "IGCM_sys_Get_Dir"
687	else
688	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
689	fi
690	fi
691	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
692	}
693
694	#D-#==================================================
695	#D-function IGCM_sys_Put_Rest
696	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
697	#D- File and target directory must exist.
698	#D-* Examples:
699	#D-
700	function IGCM_sys_Put_Rest {
701	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
702	if ( $DEBUG_sys ) ; then
703	echo "IGCM_sys_Put_Rest :" $@
704	fi
705	if [ $DRYRUN = 0 ]; then
706	if [ ! -f ${1} ] ; then
707	# The file is missing
708	if [ X${3} = X'OPTIONAL' ] ; then
709	echo "WARINING : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
710	IGCM_debug_Print 1 "IGCM_sys_Put_Rest: Optional restart file ${1} is missing, now continue for the next"
711	else
712	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
713	IGCM_debug_Exit "IGCM_sys_Put_Rest"
714	fi
715
716	else
717	# The file exist
718
719	typeset status
720	#
721	# USUAL WAY
722	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
723	status=$?
724
725	# #RSYNC WITH NETWORK SSH CALL
726	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
727	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
728
729	# #RSYNC WITH NFS USE
730	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
731	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
732
733	# status=$?
734	# IGCM_sys_Rsync_out $status
735
736	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
737	# (( status=status+$? ))
738
739	if [ ${status} -gt 0 ] ; then
740	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
741	[ -f ${1} ] && ls -l ${1}
742	[ -f ${2} ] && ls -l ${2}
743	[ -f ${2}/${1} ] && ls -l ${2}/${1}
744	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
745	IGCM_debug_Exit "IGCM_sys_Put_Rest"
746	else
747
748	if [ X${JobType} = XRUN ] ; then
749	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
750	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
751	fi
752
753	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
754	fi
755	fi
756	fi
757	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
758	}
759
760	#D-#==================================================
761	#D-function IGCM_sys_Put_Out
762	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
763	#D-* Examples:
764	#D-
765	function IGCM_sys_Put_Out {
766	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
767	if ( $DEBUG_sys ) ; then
768	echo "IGCM_sys_Put_Out :" $@
769	fi
770
771	typeset NB_ESSAI DELAI status i exist skip
772	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
773
774	# number of tentative
775	NB_ESSAI=3
776	# time delay between tentative
777	DELAI=2
778
779	if [ $DRYRUN = 0 ]; then
780	if [ ! -f ${1} ] ; then
781	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
782	IGCM_debug_PopStack "IGCM_sys_Put_Out"
783	return 1
784	fi
785	#
786	IGCM_sys_MkdirArchive $( dirname $2 )
787	#
788	exist=false
789	skip=false
790	if [ -f $2 ] ; then
791	IGCM_debug_Print 1 "$2 already exist"
792	# add here command to demigrate all offline files to reduce time of this command
793	# actually on Jean Zay files are not archive on bands
794
795	exist=true
796	if [ "X$( diff $1 $2 )" = X ] ; then
797	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
798	skip=true
799	else
800	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
801	skip=false
802	fi
803	fi
804	#
805	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
806	IGCM_sys_Chmod u+w $2
807	fi
808
809	if [ X${skip} = Xfalse ] ; then
810	i=0
811	while [ $i -lt $NB_ESSAI ] ; do
812	# Identify file system
813	fileDeviceNumberInHex=$( stat -c %d $1 )
814	status=$?
815	if [ ${status} -gt 0 ] ; then
816	IGCM_debug_Exit "IGCM_sys_Put_Out"
817	fi
818	# Identify file system
819	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
820	status=$?
821	if [ ${status} -gt 0 ] ; then
822	IGCM_debug_Exit "IGCM_sys_Put_Out"
823	fi
824
825	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
826	# They are not on the same device. USUAL WAY
827	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
828	status=$?
829	else
830	# They are on the same device. NOT SO USUAL WAY
831	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
832	status=$?
833	fi
834	if [ ${status} -gt 0 ]; then
835	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
836	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
837	[ -f ${1} ] && ls -l ${1}
838	[ -f ${2} ] && ls -l ${2}
839	[ -f ${2}/${1} ] && ls -l ${2}/${1}
840	sleep $DELAI
841	else
842	break
843	fi
844	(( i = i + 1 ))
845	done
846	fi
847
848	# #RSYNC WITH NETWORK SSH CALL
849	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
850	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
851
852	# #RSYNC WITH NFS USE
853	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
854	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
855
856	# status=$?
857	# IGCM_sys_Rsync_out $status
858
859	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
860	# (( status=status+$? ))
861
862	if [ ${status} -gt 0 ] ; then
863	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
864	[ -f ${1} ] && ls -l ${1}
865	[ -f ${2} ] && ls -l ${2}
866	[ -f ${2}/${1} ] && ls -l ${2}/${1}
867	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
868	IGCM_debug_Exit "IGCM_sys_Put_Out"
869	else
870
871	if [ X${JobType} = XRUN ] ; then
872	if [ X${3} = X ] ; then
873	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
874	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
875	fi
876	fi
877
878	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
879	fi
880	fi
881	IGCM_debug_PopStack "IGCM_sys_Put_Out"
882	return 0
883	}
884
885	#D-#==================================================
886	#D-function IGCM_sys_Get
887	#D-* Purpose: Get a file from ${ARCHIVE}
888	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
889	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
890	function IGCM_sys_Get {
891	IGCM_debug_PushStack "IGCM_sys_Get" $@
892
893	typeset DEST dm_liste target file_work
894	typeset NB_ESSAI DELAI status i
895
896	if ( $DEBUG_sys ) ; then
897	echo "IGCM_sys_Get :" $@
898	fi
899
900	# number of tentative
901	NB_ESSAI=3
902	# time delay between tentative
903	DELAI=2
904
905	if [ $DRYRUN -le 2 ]; then
906	if [ X${1} = X'/l' ] ; then
907	eval set +A dm_liste \${${2}}
908	else
909	eval set +A dm_liste ${1}
910	fi
911	eval DEST=\${${#}}
912	# add here command to demigrate all offline files to reduce time of this command
913	# actually on Jean Zay files are not archive on bands
914
915	status=$?
916	if [ ${status} -gt 0 ] ; then
917	echo "WARNING IGCM_sys_Get : error code ${status}"
918	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
919	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
920	fi
921
922	#if [ ${status} -gt 0 ] ; then
923	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
924	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
925	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
926	# sleep 30
927	# echo "We try another time"
928	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
929	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
930	# status=$?
931	# if [ ${status} -gt 0 ] ; then
932	# echo "ERROR IGCM_sys_Get : again demigration error :"
933	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
934	# IGCM_debug_Exit "IGCM_sys_Get"
935	# fi
936	# else
937	# echo "ERROR IGCM_sys_Get : demigration error :"
938	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
939	# IGCM_debug_Exit "IGCM_sys_Get"
940	# fi
941	#fi
942
943	# #RSYNC WITH NETWORK SSH CALL
944	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
945	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
946
947	# #RSYNC WITH NFS USE
948	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
949	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
950
951	# status=$?
952	# IGCM_sys_Rsync_out $status
953
954	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
955	# (( status=status+$? ))
956
957	#USUAL WAY
958	if [ X${1} = X'/l' ] ; then
959	for target in ${dm_liste[*]} ; do
960	local_file=$( basename ${target} )
961	# test if the target file is present before the loop
962	IGCM_sys_TestFileArchive ${target}
963	status=$?
964	if [ ${status} -gt 0 ] ; then
965	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
966	IGCM_debug_Exit "IGCM_sys_Get"
967	else
968	i=0
969	while [ $i -lt $NB_ESSAI ] ; do
970	#if [ X${DoLink} = Xtrue ] ; then
971	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
972	# status=$?
973	# else
974	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
975	# status=$?
976	#fi
977	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
978	status=$?
979	if [ ${status} -gt 0 ]; then
980	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
981	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
982	sleep $DELAI
983	else
984	break
985	fi
986	(( i = i + 1 ))
987	done
988	if [ ${status} -gt 0 ] ; then
989	echo "IGCM_sys_Get : error"
990	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
991	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
992	IGCM_debug_Exit "IGCM_sys_Get"
993	else
994	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
995	fi
996	fi
997	done
998	else
999	i=0
1000	while [ $i -lt $NB_ESSAI ] ; do
1001	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1002	status=$?
1003	if [ ${status} -gt 0 ]; then
1004	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
1005	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
1006	sleep $DELAI
1007	else
1008	break
1009	fi
1010	(( i = i + 1 ))
1011	done
1012	if [ ${status} -gt 0 ] ; then
1013	echo "IGCM_sys_Get : error"
1014	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1015	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1016	IGCM_debug_Exit "IGCM_sys_Get"
1017	else
1018	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1019	fi
1020	fi
1021	fi
1022	IGCM_debug_PopStack "IGCM_sys_Get"
1023	}
1024
1025	#D-#==================================================
1026	#D-function IGCM_sys_GetDate_Monitoring
1027	#D-* Purpose: get the last year for which the monitoring has been computed
1028	#D-* Examples:
1029	#D-
1030	function IGCM_sys_GetDate_Monitoring {
1031	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
1032	if ( $DEBUG_sys ) ; then
1033	echo "IGCM_sys_GetDate_Monitoring :" $@
1034	fi
1035
1036	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
1037
1038	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
1039	}
1040
1041	#D-#==================================================
1042	#D-function IGCM_sys_Dods_Rm
1043	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
1044	#D-* Examples:
1045	#D-
1046	function IGCM_sys_Dods_Rm {
1047	if ( $DEBUG_sys ) ; then
1048	echo "IGCM_sys_Dods_Rm :" $@
1049	fi
1050	typeset status
1051	if [ $DRYRUN = 0 ]; then
1052
1053	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
1054	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
1055	# echo "Nothing has been done."
1056	# return
1057	# fi
1058
1059	if [ "$#" -eq 1 ]; then
1060	thredds_rm ${R_DODS}/${1}
1061	status=$?
1062	else
1063	thredds_rm ${R_DODS}
1064	status=$?
1065	fi
1066
1067	# if [ ${status} -gt 0 ] ; then
1068	# echo "IGCM_sys_Dods_Rm : error."
1069	# cat out_dods_rm
1070	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
1071	# else
1072	# rm out_dods_rm
1073	# fi
1074
1075	fi
1076	return $status
1077	}
1078
1079	#D-#==================================================
1080	#D-function IGCM_sys_Dods_Cp
1081	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
1082	#D-* Examples:
1083	#D-
1084	function IGCM_sys_Dods_Cp {
1085	if ( $DEBUG_sys ) ; then
1086	echo "IGCM_sys_Dods_Cp :" $@
1087	fi
1088	typeset status
1089	if [ $DRYRUN = 0 ]; then
1090
1091	# if [ ! -d ${R_SAVE}/${1} ] ; then
1092	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
1093	# echo "Nothing has been done."
1094	# return
1095	# fi
1096
1097
1098	thredds_cp ${1} ./${R_DODS}/.
1099
1100	status=$?
1101
1102	# if [ ${status} -gt 0 ] ; then
1103	# echo "IGCM_sys_Dods_Cp : error."
1104	# cat out_dods_cp
1105	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1106	# else
1107	# rm out_dods_cp
1108	# fi
1109
1110	fi
1111	return $status
1112	}
1113
1114	#D-#==================================================
1115	#D-function IGCM_sys_Put_Dods
1116	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1117	#D-* Examples:
1118	#D-
1119	function IGCM_sys_Put_Dods {
1120	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1121	if ( $DEBUG_sys ) ; then
1122	echo "IGCM_sys_Put_Dods :" $@
1123	fi
1124	typeset status
1125	if [ $DRYRUN = 0 ]; then
1126	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1127	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1128	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1129	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1130	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1131	return
1132	fi
1133
1134	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1135	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1136	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1137	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1138	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1139	return
1140	fi
1141	#
1142	if [ -d ${R_SAVE}/${1} ] ; then
1143	cd ${R_SAVE}
1144	elif [ -d ${R_FIGR}/${1} ] ; then
1145	cd ${R_FIGR}
1146	fi
1147
1148	IGCM_sys_Dods_Rm ${1}
1149	IGCM_sys_Dods_Cp ${1}
1150	status=0
1151
1152	if [ ${status} -gt 0 ] ; then
1153	echo "IGCM_sys_Put_Dods : error."
1154	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1155	fi
1156	fi
1157	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1158	}
1159
1160	##############################################################
1161	# REBUILD OPERATOR
1162
1163	#D-#==================================================
1164	#D-function IGCM_sys_sync
1165	#D-* Purpose: flush buffer on disk
1166	#D-* Examples:
1167	#D-
1168	function IGCM_sys_sync {
1169	IGCM_debug_PushStack "IGCM_sys_sync" $@
1170	if ( $DEBUG_sys ) ; then
1171	echo "IGCM_sys_sync :" $@
1172	fi
1173
1174	/bin/sync
1175
1176	IGCM_debug_PopStack "IGCM_sys_sync"
1177	}
1178
1179	############################################################
1180	# Activate Running Environnment Variables
1181
1182	#D-#==================================================
1183	#D-function IGCM_sys_activ_variables
1184	#D-* Purpose: set environement variables prior to execution
1185	#D-* Examples:
1186	#D-
1187	function IGCM_sys_activ_variables {
1188	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1189	if ( $DEBUG_sys ) ; then
1190	echo "IGCM_sys_activ_variables"
1191	fi
1192
1193	# --------------------------------------------------------------------
1194	#D- MODULE specifications
1195	# --------------------------------------------------------------------
1196	# Source the file EnvFile containing all module load needed to run the model.
1197	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1198	# it will be used. Otherwise a default file will be used.
1199	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1200	# EnvFile is set in config.card and the file exists
1201	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1202	EnvFile=${config_UserChoices_EnvFile}
1203	else
1204	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1205	module purge > /dev/null 2>&1
1206	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1207	fi
1208	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1209	IGCM_debug_Print 1 ${EnvFile}
1210	. ${EnvFile}
1211
1212	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1213	module list
1214
1215	# --------------------------------------------------------------------
1216	#D- MPI specifications
1217	# --------------------------------------------------------------------
1218
1219	# --------------------------------------------------------------------
1220	#D- Other specifications
1221	# --------------------------------------------------------------------
1222
1223	ulimit -s unlimited
1224	if ( [ ${executionType} == 2 ] && [ ${job_type} == "compute" ]) ; then
1225	export SLURM_HOSTFILE=./hostlist
1226	fi
1227
1228
1229	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1230	}
1231
1232	############################################################
1233	# Desactivate Running Environnment Variables
1234
1235	#D-#==================================================
1236	#D-function IGCM_sys_desactiv_variables
1237	#D-* Purpose: unset environement variables after execution
1238	#D-* Examples:
1239	#D-
1240	function IGCM_sys_desactiv_variables {
1241	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1242	if ( $DEBUG_sys ) ; then
1243	echo "IGCM_sys_desactiv_variables"
1244	fi
1245	# --------------------------------------------------------------------
1246	#D- MPI specifications
1247	# --------------------------------------------------------------------
1248	if [ ${executionType} -eq 2 ] ; then
1249	unset SLURM_HOSTFILE
1250	fi
1251	# --------------------------------------------------------------------
1252	#D- Other specifications
1253	# --------------------------------------------------------------------
1254
1255	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1256	}
1257
1258	############################################################
1259	# Update job headers to be used by the scheduler
1260
1261	#D-#==================================================
1262	#D-function IGCM_sys_updateHeaders
1263	#D-* Purpose: Update job headers to be used by the scheduler
1264	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1265	#D-
1266	function IGCM_sys_updateHeaders {
1267	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1268	if ( $DEBUG_sys ) ; then
1269	echo "IGCM_sys_updateHeaders"
1270	fi
1271	typeset file
1272	file=$1
1273
1274	if [ ${executionType} -eq 1 ] ; then
1275	# MPMD + MPI
1276	sed -e "/::openMPthreads::/d" \
1277	-e "s/::JobNumProcTot::/${coreNumber}/" \
1278	${file} > ${file}.tmp
1279
1280	elif [ ${executionType} -eq 2 ] ; then
1281	# MPMD + MPI + OMP
1282	(( nodeNumber = coreNumber / NB_CORE_PER_NODE ))
1283	[ $(( ${coreNumber} % ${NB_CORE_PER_NODE} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
1284	sed -e "/::openMPthreads::/d" \
1285	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1286	-e "/ntasks/i\#SBATCH --nodes=${nodeNumber}"\
1287	-e "/ntasks/i\#SBATCH --exclusive" \
1288	${file} > ${file}.tmp
1289	elif [ ${executionType} -eq 3 ] ; then
1290	# SPMD + MPI/OMP
1291	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1292	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1293	${file} > ${file}.tmp
1294
1295	elif [ ${executionType} -eq 4 ] ; then
1296	# SPMD + MPI only
1297	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1298	-e "/::openMPthreads::/d" \
1299	${file} > ${file}.tmp
1300
1301	elif [ ${executionType} -eq 5 ] ; then
1302	# SPMD + OMP only
1303	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1304	-e "/::JobNumProcTot::/d" \
1305	${file} > ${file}.tmp
1306
1307	elif [ ${executionType} -eq 6 ] ; then
1308	# SEQUENTIAL THEN
1309	sed -e "s/::JobNumProcTot::/1/" \
1310	-e "/::openMPthreads::/d" \
1311	${file} > ${file}.tmp
1312
1313	fi
1314
1315	IGCM_sys_Mv ${file}.tmp ${file}
1316
1317	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1318	}
1319
1320	############################################################
1321	# Build MPI/OMP scripts run file (dummy function)
1322
1323	#D-#==================================================
1324	#D-function IGCM_sys_build_run_file
1325	#D-* Purpose: build run file (deprecated)
1326	#D-* Examples:
1327	#D-
1328	function IGCM_sys_build_run_file {
1329	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1330	}
1331
1332	############################################################
1333	# Build MPI/OMP scripts
1334
1335	#D-#==================================================
1336	#D-function IGCM_sys_build_execution_scripts
1337	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1338	#D-* Examples:
1339	#D-
1340	function IGCM_sys_build_execution_scripts
1341	{
1342	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1343	if ( $DEBUG_sys ) ; then
1344	echo "IGCM_sys_build_execution_scripts " $@
1345	fi
1346	IGCM_debug_Print 1 "executionType= ${executionType}"
1347	EXECUTION=${HOST_MPIRUN_COMMAND}
1348
1349	if ( ${OK_PARA_MPMD} ) ; then
1350
1351	# MPMD mode
1352	# 1 MPI only : executionType=1
1353	# 2 MPI/OpenMP : executionType=2
1354
1355	if [ -f run_file ] ; then
1356	IGCM_sys_Rm -f run_file
1357	fi
1358	touch run_file
1359
1360	# case 1 : Only MPI (MPMD)
1361	if ( ! ${OK_PARA_OMP} ) ; then
1362	# Build run_file
1363	current_core=0
1364	# First loop on the components for the coupler ie oasis (only if oasis3)
1365	# the coupler ie oasis3 must be the first one
1366	for comp in ${config_ListOfComponents[*]} ; do
1367
1368	eval ExeNameIn=\${config_Executable_${comp}[0]}
1369	eval ExeNameOut=\${config_Executable_${comp}[1]}
1370
1371	# Only if the component has an executable
1372	if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1373
1374	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1375	(( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
1376	echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
1377	(( current_core = ${end_core} + 1 ))
1378	fi
1379	done
1380
1381	EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
1382
1383	IGCM_sys_Chmod u+x run_file
1384	if ( $DEBUG_sys ) ; then
1385	echo "run_file contains : "
1386	cat run_file
1387	fi
1388
1389	else
1390
1391	# 2 MPI/OpenMP : executionType=2
1392
1393	# MPI-OpenMP (MPMD)
1394	# export SLURM_HOSTFILE=./hostlist
1395	# srun --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file
1396	# example of hostlist file :
1397	# r3i3n33
1398	# r3i3n33
1399	# ...
1400	# example of run_file :
1401	# 0-70 ./prog_lmdz.x.sh %o %t
1402	# 71-430 ./prog_opa.xx.sh %o %t
1403	# 431-431 ./prog_xios.x.sh %o %t
1404	# examples of prog_file :
1405	# prog_lmdz.x.sh :
1406	# (( init = 0 + $1 ))
1407	# (( index = init * 10 ))
1408	# (( slot = index % 40 ))
1409	# taskset -c $slot-$((slot + 10 - 1)) ./script_lmdz.x.ksh
1410	# that will become
1411	# taskset -c 0-9 ./script_lmdz.x.ksh
1412	# ...
1413	# with script_lmdz.x.ksh
1414	# export OMP_STACKSIZE=3g
1415	# export OMP_PLACES=cores
1416	# export OMP_NUM_THREADS=10
1417	# ./lmdz.x > out_lmdz.x.out.${SLURM_PROCID} 2>out_lmdz.x.err.${SLURM_PROCID}
1418
1419	# Hosts treatment
1420	_bkIFS=$IFS;
1421	IFS=$'\n'; set -f
1422	listnodes=($(< <( scontrol show hostnames $SLURM_JOB_NODELIST )))
1423	IFS=$_bkIFS; set +f
1424	rm -f hostlist
1425
1426	# Loop on the components to build run_file and script_exec files
1427	rank=0
1428	current_core=0
1429	current_core_mpi=0
1430
1431	for comp in ${config_ListOfComponents[*]} ; do
1432
1433	eval ExeNameIn=\${config_Executable_${comp}[0]}
1434	eval ExeNameOut=\${config_Executable_${comp}[1]}
1435
1436	# Not possible if oasis has an executable (i.e old version of oasis3)
1437	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1438	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1439	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1440	IGCM_debug_Verif_Exit
1441	fi
1442
1443	# Only if we really have an executable for the component :
1444	if [ "X${ExeNameOut}" != X\"\" ] ; then
1445
1446	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1447	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1448	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1449
1450
1451	# Build script files
1452
1453	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1454	echo "" >> script_${ExeNameOut}.ksh
1455	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1456	# Check if the number of threads is correct
1457	case ${comp_proc_omp_loc} in
1458	2\|4\|5\|10\|20)
1459	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1460	;;
1461	*)
1462	IGCM_debug_Exit "ERROR with OMP parameters !"
1463	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1464	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1465	IGCM_debug_Verif_Exit
1466	;;
1467	esac
1468	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1469	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
1470	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1471
1472	fi
1473
1474	# to have out/err per process on different files
1475	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1476
1477	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1478
1479	# Build run_file
1480	# Only if the component has an executable
1481	if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1482
1483	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1484	(( end_core = ${current_core_mpi} + ${comp_proc_mpi_loc} - 1 ))
1485	echo "${current_core_mpi}-${end_core} ./prog_${ExeNameOut}.sh %o %t" >> run_file
1486	(( current_core_mpi = ${end_core} + 1 ))
1487	fi
1488
1489	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1490	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1491	else
1492	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1493	fi
1494
1495	# Build configuration file
1496
1497	echo "#!/bin/sh" > prog_${ExeNameOut}.sh
1498	echo "(( init = $current_core + \$1 ))" >> prog_${ExeNameOut}.sh
1499	echo "(( index = init * $comp_proc_omp_loc ))" >> prog_${ExeNameOut}.sh
1500	echo "(( slot = index % 40 ))" >> prog_${ExeNameOut}.sh
1501	echo "echo ${ExeNameOut} taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1))" >> prog_${ExeNameOut}.sh
1502	echo "taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1)) ./script_${ExeNameOut}.ksh" >> prog_${ExeNameOut}.sh
1503
1504	IGCM_sys_Chmod u+x prog_${ExeNameOut}.sh
1505
1506	# Build hostlist file
1507
1508	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1509	(( index_host = current_core / NB_CORE_PER_NODE ))
1510	host_value=${listnodes[${index_host}]}
1511	if [ ${job_type} == "prerun" ] ; then
1512	echo "node_${index_host}_X" >> hostlist_template
1513	else
1514	echo "$host_value" >> hostlist
1515	fi
1516	(( current_core = current_core + offset_comp_proc_loc ))
1517	done
1518	fi
1519	done
1520
1521	## variable added to stop after 60s instead of 600s by default.
1522	## This is used when no error comes from executables and when something stopped an executable without notice.
1523	export SLURM_WAIT=60
1524
1525	EXECUTION="${HOST_MPIRUN_COMMAND} --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file"
1526
1527	IGCM_sys_Chmod u+x run_file
1528	if ( $DEBUG_sys ) ; then
1529	echo "run_file contains : "
1530	cat run_file
1531	fi
1532
1533	fi # if ${OK_PARA_MPMD}
1534
1535	else
1536	# Only one executable (SPMD mode): executionType=3, 4, 5 and 6
1537
1538	for comp in ${config_ListOfComponents[*]} ; do
1539
1540	# Only if we really have an executable for the component :
1541	eval ExeNameOut=\${config_Executable_${comp}[1]}
1542	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1543
1544	# Build script files
1545
1546	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1547	echo "" >> script_${ExeNameOut}.ksh
1548	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1549
1550	if ( ${OK_PARA_OMP} ) ; then
1551	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1552	# Check if the number of threads is correct
1553	case ${comp_proc_omp_loc} in
1554	2\|4\|5\|10\|20)
1555	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1556	;;
1557	*)
1558	IGCM_debug_Exit "ERROR with OMP parameters !"
1559	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1560	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1561	IGCM_debug_Verif_Exit
1562	;;
1563	esac
1564	echo "" >> script_${ExeNameOut}.ksh
1565	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1566	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
1567	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1568	fi
1569
1570	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1571
1572	# To have out/err per process on different files
1573	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1574	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1575
1576	IGCM_debug_Print 1 "sys Jean-Zay : script_${ExeNameOut}.ksh contains"
1577	cat script_${ExeNameOut}.ksh
1578
1579	fi
1580
1581	done
1582
1583	fi # ${OK_PARA_MPMD}
1584
1585	IGCM_debug_Print 1 "sys Jean-Zay : execution command is "
1586	IGCM_debug_Print 1 "$EXECUTION"
1587
1588	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1589	}
1590
1591	#D-#==================================================
1592	#D-function IGCM_sys_check_path
1593	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1594	#D-* do not point to an important use directory. Stop immediately in that case.
1595	#D-* Examples:
1596	#D-
1597	function IGCM_sys_check_path {
1598	IGCM_debug_PushStack "IGCM_sys_check_path"
1599	if ( $DEBUG_sys ) ; then
1600	echo "IGCM_sys_check_path"
1601	fi
1602
1603	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1604	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1605	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1606	IGCM_debug_Exit "This will stop the job"
1607	elif ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X/gpfswork/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/lustre/fsn1/projects/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/gpfsstore/rech/${DataProject}/${LOGIN} ] ) ; then
1608	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1609	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1610	IGCM_debug_Exit "This will stop the job"
1611	fi
1612	IGCM_debug_PopStack "IGCM_sys_check_path"
1613	}
1614
1615	#D-#==================================================
1616	#D-function IGCM_sys_check_quota
1617	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1618	#D-* Examples:
1619	#D-
1620	function IGCM_sys_check_quota {
1621	IGCM_debug_PushStack "IGCM_sys_check_quota"
1622	echo "IGCM_sys_check_quota is desactivated on JeanZay because there is actually no way to check scratch quota on this computing center (2019.11.19)"
1623	# if ( $DEBUG_sys ) ; then
1624	# echo "IGCM_sys_check_quota"
1625	# fi
1626	# # Limit of quota (in %)
1627	# limit_quota=90
1628	#
1629	# # Check of the volume
1630	# volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1631	# volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1632	#
1633	# if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1634	#
1635	# unit_avail=${volume_avail: -1}
1636	# unit_quota=${volume_quota: -1}
1637	#
1638	# if [ "${unit_quota}" = "*" ] ; then
1639	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1640	# IGCM_debug_Print 1 "More than 100% of your quota is used"
1641	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1642	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1643	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1644	# IGCM_debug_Verif_Exit
1645	# fi
1646	#
1647	# temp_avail=${volume_avail%%${volume_avail: -1}*}
1648	# temp_quota=${volume_quota%%${volume_quota: -1}*}
1649	#
1650	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
1651	#
1652	# # Convertion
1653	# if [ ${volume_avail: -1} = "T" ] ; then
1654	# (( temp_avail = temp_avail * 1000000000000 ))
1655	# elif [ ${volume_avail: -1} = "G" ] ; then
1656	# (( temp_avail = temp_avail * 1000000000 ))
1657	# elif [ ${volume_avail: -1} = "M" ] ; then
1658	# (( temp_avail = temp_avail * 1000000 ))
1659	# elif [ ${volume_avail: -1} = "k" ] ; then
1660	# (( temp_avail = temp_avail * 1000 ))
1661	# else
1662	# (( temp_avail = volume_avail ))
1663	# fi
1664	# if [ ${volume_quota: -1} = "T" ] ; then
1665	# (( temp_quota = temp_quota * 1000000000000 ))
1666	# elif [ ${volume_quota: -1} = "G" ] ; then
1667	# (( temp_quota = temp_quota * 1000000000 ))
1668	# elif [ ${volume_quota: -1} = "M" ] ; then
1669	# (( temp_quota = temp_quota * 1000000 ))
1670	# elif [ ${volume_quota: -1} = "k" ] ; then
1671	# (( temp_quota = temp_quota * 1000 ))
1672	# else
1673	# (( temp_quota = volume_quota ))
1674	# fi
1675	# fi
1676	#
1677	# quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1678	## echo "volume ratio is " $quota_volume
1679	#
1680	# if [ ${quota_volume} -ge ${limit_quota} ] ; then
1681	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1682	# IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1683	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1684	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1685	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1686	# IGCM_debug_Verif_Exit
1687	# fi
1688	#
1689	# fi
1690	#
1691	## Check of the number of inodes
1692	#
1693	# inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1694	# inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1695	#
1696	# if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1697	#
1698	# unit_avail=${inode_avail: -1}
1699	# unit_quota=${inode_quota: -1}
1700	#
1701	# if [ "${unit_quota}" = "*" ] ; then
1702	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1703	# IGCM_debug_Print 1 "More than 100% of your quota is used"
1704	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1705	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1706	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1707	# IGCM_debug_Verif_Exit
1708	# fi
1709	#
1710	# temp_avail=${inode_avail%%${inode_avail: -1}*}
1711	# temp_quota=${inode_quota%%${inode_quota: -1}*}
1712	#
1713	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
1714	#
1715	# # Convertion
1716	# if [ ${inode_avail: -1} = "T" ] ; then
1717	# (( temp_avail = temp_avail * 1000000000000 ))
1718	# elif [ ${inode_avail: -1} = "G" ] ; then
1719	# (( temp_avail = temp_avail * 1000000000 ))
1720	# elif [ ${inode_avail: -1} = "M" ] ; then
1721	# (( temp_avail = temp_avail * 1000000 ))
1722	# elif [ ${inode_avail: -1} = "k" ] ; then
1723	# (( temp_avail = temp_avail * 1000 ))
1724	# else
1725	# (( temp_avail = inode_avail ))
1726	# fi
1727	#
1728	# if [ ${inode_quota: -1} = "T" ] ; then
1729	# (( temp_quota = temp_quota * 1000000000000 ))
1730	# elif [ ${inode_quota: -1} = "G" ] ; then
1731	# (( temp_quota = temp_quota * 1000000000 ))
1732	# elif [ ${inode_quota: -1} = "M" ] ; then
1733	# (( temp_quota = temp_quota * 1000000 ))
1734	# elif [ ${inode_quota: -1} = "k" ] ; then
1735	# (( temp_quota = temp_quota * 1000 ))
1736	# else
1737	# (( temp_quota = inode_quota ))
1738	# fi
1739	# fi
1740	# quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1741	## echo "inode ratio is " $quota_inode
1742	#
1743	# if [ ${quota_inode} -ge ${limit_quota} ] ; then
1744	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1745	# IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1746	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
1747	# IGCM_debug_Print 1 "You must have more than 10% available to run"
1748	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1749	# IGCM_debug_Verif_Exit
1750	# fi
1751	# fi
1752	IGCM_debug_PopStack "IGCM_sys_check_quota"
1753	}
1754
1755	#D-#==================================================
1756	#D-function IGCM_sys_projectAccounting
1757	#D-* Purpose: store project accounting information in a file
1758	#D-* Examples:
1759	#D-
1760	function IGCM_sys_projectAccounting {
1761	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1762	if ( $DEBUG_sys ) ; then
1763	echo "IGCM_sys_check_quota"
1764	fi
1765	#need to implemented on Jean Zay
1766	touch $1
1767	# ssh irene191 /usr/bin/ccc_myproject > $1
1768
1769	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1770	}
1771
1772	#D-#==================================================
1773	#D-function IGCM_sys_getJobSchedulerID
1774	#D-* Purpose: Get the job ID during execution
1775	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1776	#D-
1777	function IGCM_sys_getJobSchedulerID {
1778	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1779	if ( $DEBUG_sys ) ; then
1780	echo "IGCM_sys_getJobSchedulerID"
1781	fi
1782
1783	eval ${1}=${SLURM_JOBID}
1784
1785	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1786	}
1787
1788	#D-#==================================================
1789	#D-function IGCM_sys_GetJobID
1790	#D-* Purpose: Get the job ID from the JobName
1791	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1792	#D-
1793	function IGCM_sys_GetJobID {
1794	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1795	if ( $DEBUG_sys ) ; then
1796	echo "IGCM_sys_GetJobID"
1797	fi
1798
1799	ID=$( squeue -u $user \| grep Job_${config_UserChoices_JobName} \| gawk ' { print $1}' )
1800
1801
1802	eval ${3}=${ID}
1803	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1804	}
1805
1806	#D-#==================================================
1807	#D-function IGCM_sys_CountJobInQueue
1808	#D-* Purpose: Count number of users job
1809	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1810	#D-
1811	function IGCM_sys_CountJobInQueue {
1812	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1813	if ( $DEBUG_sys ) ; then
1814	echo "IGCM_sys_CountJobInQueue"
1815	fi
1816
1817	# With -f option, the full job name is given in the last column
1818	NbRun=$(squeue -u $user \| \
1819	grep -v JOBID \| gawk 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1820
1821	eval ${2}=${NbRun}
1822
1823	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1824	}
1825
1826	#D-#==================================================
1827	#D-function IGCM_sys_ListJobInQueue
1828	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1829	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1830	#D-
1831	function IGCM_sys_ListJobInQueue {
1832	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1833	if ( $DEBUG_sys ) ; then
1834	echo "IGCM_sys_ListJobInQueue"
1835	fi
1836
1837	# to be implemented on Jean Zay
1838	set -A JobList $( squeue -u $user \| gawk '{print $3}' \| grep -v NAME)
1839	# # With -f option, the full job name is given in the last column
1840	# set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1841	# '( $2 == User && \
1842	# $NF != /TS/ && \
1843	# $NF !~ /PACK/ && \
1844	# $NF !~ /REBUILD/ && \
1845	# $NF !~ /pack/ ) \
1846	# { print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1847	#
1848	eval set -A ${2} ${JobList[*]}
1849
1850	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1851	}
1852
1853	#D-#==================================================
1854	#D-function IGCM_sys_atlas
1855	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1856	#D-* Examples:
1857	#D-
1858	function IGCM_sys_atlas {
1859	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1860	if ( $DEBUG_sys ) ; then
1861	echo "IGCM_sys_atlas :" $@
1862	fi
1863
1864	typeset status
1865	# To be implemented on Jean Zay
1866	#
1867	# \ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1868	# status=$?
1869	# if [ ${status} -gt 0 ] ; then
1870	# echo "IGCM_sys_atlas : error code ${status}"
1871	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1872	# \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1873	# IGCM_debug_PopStack "IGCM_sys_atlas"
1874	# return 1
1875	# else
1876	# IGCM_debug_PopStack "IGCM_sys_atlas"
1877	# return 0
1878	# fi
1879	#
1880	IGCM_debug_PopStack "IGCM_sys_atlas"
1881	}
1882
1883	#D-#==================================================
1884	#D-function IGCM_sys_rebuild_nemo
1885	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1886	#D-* Examples:
1887	#D-
1888
1889	function IGCM_sys_rebuild_nemo {
1890	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1891	if ( $DEBUG_sys ) ; then
1892	echo "IGCM_sys_rebuild_nemo :" $@
1893	fi
1894
1895	/gpfswork/rech/psl/commun/Tools/rebuild_nemo/bin/rebuild_nemo ${1} ${2}
1896
1897	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1898	}
1899
1900	#
1901	#D-#==================================================
1902	#D-function IGCM_sys_launch_job_compute
1903	#D-* Purpose: launch the computing part of the simulation on computing partition
1904	#D-* Examples:
1905	#D-
1906	function IGCM_sys_launch_job_compute {
1907	IGCM_debug_PushStack "IGCM_sys_launch_job_compute"
1908	if ( $DEBUG_sys ) ; then
1909	echo "IGCM_sys_launch_job_compute :"
1910	fi
1911	if [ -f ${libIGCM}/Job_compute_${config_UserChoices_JobName} ] ; then
1912
1913	SUBMIT_DIRECTORY=$(grep SUBMIT_DIR ${libIGCM}/libIGCM_sys/libIGCM_sys_${SYSTEM}.ksh \| grep -m1 typeset \| cut -dx -f2)
1914	sed -e "s%::EXECUTION::%${EXECUTION}%g" \
1915	-e "s%::EXECUTION_TYPE::%${executionType}%" \
1916	-e "s%::JOBNAME::%${config_UserChoices_JobName}%" \
1917	${libIGCM}/Job_compute_${config_UserChoices_JobName} > ${RUN_DIR}/Job_compute_${config_UserChoices_JobName}
1918	cp ${EnvFile} ${RUN_DIR}/.
1919	cd ${RUN_DIR} ; IGCM_sys_Qsub_job_compute Job_compute_${config_UserChoices_JobName}
1920	IGCM_debug_Verif_Exit
1921	echo
1922	echo "############################################"
1923	echo "# END OF PRERUN STEP #"
1924	echo "############################################"
1925	echo
1926	exit
1927	else
1928	IGCM_debug_Exit "ERROR with computing job !"
1929	IGCM_debug_Print 2 "Computing job is missing"
1930	IGCM_debug_Verif_Exit
1931	fi
1932
1933	IGCM_debug_PopStack "IGCM_sys_launch_job_compute"
1934	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: