Context Navigation

source: branches/libIGCM_CESMEP/libIGCM_sys/libIGCM_sys_irene.ksh @ 1564

Last change on this file since 1564 was 1563, checked in by ssenesi, 19 months ago
Better gues default_project on irene(-amd) in libIGCM_sys
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 63.9 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Irene
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=irene
71	# add default project on irene
72	set -x
73	if [ ! -z $BRIDGE_MSUB_PROJECT ]; then
74	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT} \| cut -d@ -f1 )
75	else
76	typeset PROJECT=$(echo ${CCCHOME} \| cut -d/ -f5 )
77	fi
78	set +x
79	# jobWarningDelay in seconds
80	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
81
82	#D-
83	#D-#==================================================
84	#D-Program used in libIGCM
85	#D-#==================================================
86
87	# Submit command
88	typeset SUBMIT=${SUBMIT:=ccc_msub}
89	# rsync with path
90	typeset -r RSYNC=/usr/bin/rsync
91	# RSYNC_opt args to rsync
92	typeset -r RSYNC_opt="-va"
93	# ie storage filesystem
94	typeset -r STOREHOST=${MASTER}
95	typeset -r REMOTE_RSYNC=/usr/bin/rsync
96
97	#====================================================
98	# Set environment tools (ferret, nco, cdo, rebuild, ...)
99	#====================================================
100	if [ X${TaskType} = Xcomputing ] ; then
101	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
102	else
103	module purge > /dev/null 2>&1
104	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
105	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
106	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
107	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
108
109	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
110	export UVCDAT_ANONYMOUS_LOG=FALSE
111	fi
112
113	# FYI
114	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
115	[ ! X${TaskType} = Xchecking ] && module list
116
117	# For AddNoise and AddPertu3DOCE
118	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
119	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
120
121	# For rebuild
122	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
123	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
124
125	#====================================================
126	# Host specific DIRECTORIES
127	#====================================================
128
129	# ============ CESIUM START ============ #
130
131	#====================================================
132	#- Mirror libIGCM from titane to cesium if needed
133	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
134	#if [ ! ${ROOTSYS} = "home" ] ; then
135	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
136	#else
137	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
138	#fi
139
140	#====================================================
141	#- libIGCM_POST
142	#if ( ${MirrorlibIGCM} ) ; then
143	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
144	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
145	#else
146	# typeset -r libIGCM_POST=${libIGCM}
147	#fi
148
149	# ============ CESIUM END ============ #
150
151	#====================================================
152	#- MirrorlibIGCM for frontend
153	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
154
155	#====================================================
156	#- libIGCM_POST for frontend
157	typeset -r libIGCM_POST=${libIGCM}
158
159	#====================================================
160	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
161	typeset -r R_EXE="${MODIPSL}/bin"
162
163	#====================================================
164	#- SUBMIT_DIR : submission dir
165	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
166
167	#====================================================
168	#- IN
169	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
170
171	#====================================================
172	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
173	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
174
175	#====================================================
176	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
177	typeset -r OUTCOMMAND_PATH=/tmp
178
179	#====================================================
180	#- HOST_MPIRUN_COMMAND
181	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
182
183	#====================================================
184	#- Max number of arguments passed to nco operator or demigration command
185	UNIX_MAX_LIMIT=360
186
187	#====================================================
188	#- set PackDefault to true on irene
189	PackDefault=true
190
191	#====================================================
192	#- Default number of MPI task for IPSL coupled model
193	#- required for backward compatibility
194	#-
195	DEFAULT_NUM_PROC_OCE=5
196	DEFAULT_NUM_PROC_CPL=1
197	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
198
199	#====================================================
200	#- Number of computing cores per node
201	#-
202	NB_CORE_PER_NODE=48
203
204	#D-#==================================================
205	#D-function IGCM_sys_defineArchives
206	#D-* Purpose:
207	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
208	#D-* Define ARCHIVE : Dedicated to large files
209	#D-* Define STORAGE : Dedicated to small/medium files
210	#D-* Define R_OUT : Output tree located on ARCHIVE
211	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
212	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
213	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
214	#D-* Examples:
215	#D-
216	function IGCM_sys_defineArchives {
217	IGCM_debug_PushStack "IGCM_sys_defineArchives"
218	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
219	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = XDEFAULT ]; then
220	# Default option: Change the dfldatadir according to the project used for submission
221	# The variable DataProject in section UserChoices in config.card is not set or is set to DEFAULT
222	if [ X"$( echo ${PROJECT} \| grep cmip6 )" != "X" ] ; then
223	module switch dfldatadir dfldatadir/gencmip6
224	IGCM_debug_Print 1 "Load dfldatadir for forced gencmip6 project"
225	else
226	echo "Load dfldatadir for project ${PROJECT}"
227	module switch dfldatadir dfldatadir/${PROJECT}
228	IGCM_debug_Print 1 "Load dfldatadir for project ${PROJECT}"
229	fi
230	else
231	# Use the project set by the variable DataProject in section UserChoices in config.card
232	module switch dfldatadir dfldatadir/${config_UserChoices_DataProject}
233	IGCM_debug_Print 1 "Load dfldatadir for DataProject read in config.card : ${config_UserChoices_DataProject}"
234	fi
235	#====================================================
236	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
237	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
238
239	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
240	#====================================================
241	#- ARCHIVE (dedicated to large files)
242	ARCHIVE=${config_UserChoices_ARCHIVE}
243	else
244	#====================================================
245	#- ARCHIVE (dedicated to large files)
246	ARCHIVE=${CCCSTOREDIR}
247	fi
248
249	if [ ! X${config_UserChoices_STORAGE} = X ]; then
250	#====================================================
251	#- STORAGE (dedicated to small/medium files)
252	STORAGE=${config_UserChoices_STORAGE}
253	else
254	#====================================================
255	#- STORAGE (dedicated to small/medium files)
256	STORAGE=${CCCWORKDIR}
257	fi
258
259	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
260	#====================================================
261	#- R_OUT
262	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
263
264	#====================================================
265	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
266	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
267
268	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
269
270	else
271	#====================================================
272	#- R_OUT
273	R_OUT=${ARCHIVE}/IGCM_OUT
274
275	#====================================================
276	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
277	R_FIG=${STORAGE}/IGCM_OUT
278	fi
279
280	#====================================================
281	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
282	CMIP6_BUF=${STORAGE}/IGCM_OUT
283
284	#====================================================
285	#- R_BUF (ONLY FOR double copy an scratch)
286	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
287
288	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
289	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
290	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
291
292	IGCM_debug_PopStack "IGCM_sys_defineArchives"
293	}
294
295	#D-#==================================================
296	#D-function IGCM_sys_RshArchive
297	#D-* Purpose: Archive rsh command
298	#D-* Examples:
299	#D-
300	function IGCM_sys_RshArchive {
301	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
302	/bin/ksh <<-EOF
303	${@}
304	EOF
305	status=$?
306	if [ ${status} -gt 0 ] ; then
307	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
308	IGCM_debug_Exit "IGCM_sys_RshArchive"
309	fi
310	IGCM_debug_PopStack "IGCM_sys_RshArchive"
311	}
312
313	#D-#==================================================
314	#D-function IGCM_sys_RshArchive_NoError
315	#D-* Purpose: Archive rsh command, without error
316	#D-* used only in monitoring.job
317	#D-* Examples:
318	#D-
319	function IGCM_sys_RshArchive_NoError {
320	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
321	/bin/ksh <<-EOF
322	${@} 2> /dev/null
323	EOF
324	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
325	}
326
327	#D-#==================================================
328	#D-function IGCM_sys_MkdirArchive
329	#D-* Purpose: Mkdir on Archive
330	#D-* Examples:
331	#D-
332	function IGCM_sys_MkdirArchive {
333	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
334	if ( $DEBUG_sys ) ; then
335	echo "IGCM_sys_MkdirArchive :" $@
336	fi
337	#- creation de repertoire sur le serveur fichier
338	if [ ! -d ${1} ]; then
339	\mkdir -p $1
340	status=$?
341
342	if [ ${status} -gt 0 ] ; then
343	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
344	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
345	fi
346	fi
347	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
348	}
349
350	#D-#==================================================
351	#D-function IGCM_sys_TestDirArchive
352	#D-* Purpose: Test Directory that must exists on Archive
353	#D-* Examples:
354	#D-
355	function IGCM_sys_TestDirArchive {
356	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
357	if ( $DEBUG_sys ) ; then
358	echo "IGCM_sys_TestDirArchive :" $@
359	fi
360	typeset ExistFlag
361	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
362	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
363	return ${ExistFlag}
364	}
365
366	#D-#==================================================
367	#D-function IGCM_sys_IsFileArchived
368	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
369	#D-* Examples:
370	#D-
371	function IGCM_sys_IsFileArchived {
372	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
373	if ( $DEBUG_sys ) ; then
374	echo "IGCM_sys_IsFileArchived :" $@
375	fi
376	typeset IsArchivedFlag
377	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
378	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
379
380	return ${IsArchivedFlag}
381	}
382
383	#D-#==================================================
384	#D-function IGCM_sys_TestFileArchive
385	#D-* Purpose: Test file that must NOT EXISTS on Archive
386	#D-* Examples:
387	#D-
388	function IGCM_sys_TestFileArchive {
389	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
390	typeset ExistFlag
391	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
392	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
393
394	return ${ExistFlag}
395	}
396
397	#D-#==================================================
398	#D-function IGCM_sys_CountFileArchive
399	#D-* Purpose: Count files on Archive filesystem
400	#D-* Examples:
401	#D-
402	function IGCM_sys_CountFileArchive {
403	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
404	ls ${@} 2>/dev/null \| wc -l
405	if [ $? -gt 0 ] ; then
406	echo "IGCM_sys_CountFileArchive : erreur."
407	fi
408	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
409	}
410
411	#D-#==================================================
412	#D-function IGCM_sys_Tree
413	#D-* Purpose: Tree directories with files on ${ARCHIVE}
414	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
415	#D-
416	function IGCM_sys_Tree {
417	IGCM_debug_PushStack "IGCM_sys_Tree" $@
418	if ( $DEBUG_sys ) ; then
419	echo "IGCM_sys_Tree :" $@
420	fi
421
422	\ls -lR ${@}
423
424	IGCM_debug_PopStack "IGCM_sys_Tree"
425	}
426
427	#D-#==================================================
428	#D-function IGCM_sys_Qsub
429	#D-* Purpose: Qsub new job
430	#D-* Examples:
431	#D-
432	function IGCM_sys_Qsub {
433	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
434	if ( $DEBUG_sys ) ; then
435	echo "IGCM_sys_Qsub :" $@
436	fi
437	typeset options status
438	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
439
440	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
441	status=$?
442
443	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
444	if [ ${status} -gt 0 ] ; then
445	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
446	IGCM_debug_Exit "IGCM_sys_Qsub"
447	else
448	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
449	fi
450	IGCM_debug_PopStack "IGCM_sys_Qsub"
451	}
452
453	#D-#==================================================
454	#D-function IGCM_sys_QsubPost
455	#D-* Purpose: Qsub new job on scalaire
456	#D-* Examples:
457	#D-
458	function IGCM_sys_QsubPost {
459	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
460	if ( $DEBUG_sys ) ; then
461	echo "IGCM_sys_QsubPost :" $@
462	fi
463	typeset options status
464	options="-Q normal -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
465
466	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
467	status=$?
468
469	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
470	if [ ${status} -gt 0 ] ; then
471	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
472	IGCM_debug_Exit "IGCM_sys_QsubPost"
473	else
474	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
475	fi
476	IGCM_debug_PopStack "IGCM_sys_QsubPost"
477	}
478
479	#D-*************************
480	#D- File transfer functions
481	#D-*************************
482	#D-
483
484	#D-#==================================================
485	#D-function IGCM_sys_RmRunDir
486	#D-* Purpose: rm tmpdir (dummy function most of the time batch
487	#D- scheduler will do the job)
488	#D-* Examples:
489	#D-
490	function IGCM_sys_RmRunDir {
491	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
492	if ( $DEBUG_sys ) ; then
493	echo "IGCM_sys_RmRunDir :" $@
494	fi
495
496	typeset status
497
498	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
499	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
500	status=$?
501
502	if [ ${status} -gt 0 ] ; then
503	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
504	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
505	IGCM_debug_Exit "IGCM_sys_RmRunDir"
506	else
507	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
508	fi
509	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
510	}
511
512	#D-#==================================================
513	#D-function IGCM_sys_Put_Dir
514	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
515	#D-* Examples:
516	#D-
517	function IGCM_sys_Put_Dir {
518	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
519	if ( $DEBUG_sys ) ; then
520	echo "IGCM_sys_Put_Dir :" $@
521	fi
522	if [ $DRYRUN = 0 ]; then
523	if [ ! -d ${1} ] ; then
524	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
525	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
526	return
527	fi
528
529	typeset status
530
531	# Only if we use rsync
532	#IGCM_sys_TestDirArchive $( dirname $2 )
533	#
534	#USUAL WAY
535	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
536	status=$?
537
538	if [ ${status} -gt 0 ] ; then
539	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
540	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
541	IGCM_debug_Exit "IGCM_sys_Put_Dir"
542	else
543	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
544	fi
545	fi
546	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
547	}
548
549	#D-#==================================================
550	#D-function IGCM_sys_Get_Dir
551	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
552	#D-* Examples:
553	#D-
554	function IGCM_sys_Get_Dir {
555	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
556	if ( $DEBUG_sys ) ; then
557	echo "IGCM_sys_Get_Dir :" $@
558	fi
559	if [ $DRYRUN = 0 ]; then
560	typeset NB_ESSAI DELAI status i
561	# number of tentative
562	NB_ESSAI=3
563	# time delay between tentative
564	DELAI=2
565
566	#
567	# USUAL WAY
568	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
569	ccc_hsm get -r $1
570
571	i=0
572	while [ $i -lt $NB_ESSAI ] ; do
573	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
574	status=$?
575	if [ ${status} -gt 0 ] ; then
576	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
577	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
578	sleep $DELAI
579	else
580	break
581	fi
582	(( i = i + 1 ))
583	done
584
585	if [ ${status} -gt 0 ] ; then
586	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
587	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
588	IGCM_debug_Exit "IGCM_sys_Get_Dir"
589	else
590	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
591	fi
592	fi
593	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
594	}
595
596	#D-#==================================================
597	#D-function IGCM_sys_Put_Rest
598	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
599	#D- File and target directory must exist.
600	#D-* Examples:
601	#D-
602	function IGCM_sys_Put_Rest {
603	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
604	if ( $DEBUG_sys ) ; then
605	echo "IGCM_sys_Put_Rest :" $@
606	fi
607	if [ $DRYRUN = 0 ]; then
608	if [ ! -f ${1} ] ; then
609	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
610	IGCM_debug_Exit "IGCM_sys_Put_Rest"
611	fi
612
613	typeset status
614	#
615	# USUAL WAY
616	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
617	status=$?
618
619	# #RSYNC WITH NETWORK SSH CALL
620	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
621	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
622
623	# #RSYNC WITH NFS USE
624	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
625	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
626
627	# status=$?
628	# IGCM_sys_Rsync_out $status
629
630	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
631	# (( status=status+$? ))
632
633	if [ ${status} -gt 0 ] ; then
634	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
635	[ -f ${1} ] && ls -l ${1}
636	[ -f ${2} ] && ls -l ${2}
637	[ -f ${2}/${1} ] && ls -l ${2}/${1}
638	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
639	IGCM_debug_Exit "IGCM_sys_Put_Rest"
640	else
641
642	if [ X${JobType} = XRUN ] ; then
643	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
644	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
645	fi
646
647	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
648	fi
649	fi
650	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
651	}
652
653	#D-#==================================================
654	#D-function IGCM_sys_Put_Out
655	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
656	#D-* Examples:
657	#D-
658	function IGCM_sys_Put_Out {
659	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
660	if ( $DEBUG_sys ) ; then
661	echo "IGCM_sys_Put_Out :" $@
662	fi
663
664	typeset NB_ESSAI DELAI status i exist skip
665	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
666
667	# number of tentative
668	NB_ESSAI=3
669	# time delay between tentative
670	DELAI=2
671
672	if [ $DRYRUN = 0 ]; then
673	if [ ! -f ${1} ] ; then
674	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
675	IGCM_debug_PopStack "IGCM_sys_Put_Out"
676	return 1
677	fi
678	#
679	IGCM_sys_MkdirArchive $( dirname $2 )
680	#
681	exist=false
682	skip=false
683	if [ -f $2 ] ; then
684	IGCM_debug_Print 1 "$2 already exist"
685	ccc_hsm get $2
686	exist=true
687	if [ "X$( diff $1 $2 )" = X ] ; then
688	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
689	skip=true
690	else
691	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
692	skip=false
693	fi
694	fi
695	#
696	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
697	IGCM_sys_Chmod u+w $2
698	fi
699
700	if [ X${skip} = Xfalse ] ; then
701	i=0
702	while [ $i -lt $NB_ESSAI ] ; do
703	# Identify file system
704	fileDeviceNumberInHex=$( stat -c %d $1 )
705	status=$?
706	if [ ${status} -gt 0 ] ; then
707	IGCM_debug_Exit "IGCM_sys_Put_Out"
708	fi
709	# Identify file system
710	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
711	status=$?
712	if [ ${status} -gt 0 ] ; then
713	IGCM_debug_Exit "IGCM_sys_Put_Out"
714	fi
715
716	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
717	# They are not on the same device. USUAL WAY
718	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
719	status=$?
720	else
721	# They are on the same device. NOT SO USUAL WAY
722	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
723	status=$?
724	fi
725	if [ ${status} -gt 0 ]; then
726	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
727	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
728	[ -f ${1} ] && ls -l ${1}
729	[ -f ${2} ] && ls -l ${2}
730	[ -f ${2}/${1} ] && ls -l ${2}/${1}
731	sleep $DELAI
732	else
733	break
734	fi
735	(( i = i + 1 ))
736	done
737	fi
738
739	# #RSYNC WITH NETWORK SSH CALL
740	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
741	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
742
743	# #RSYNC WITH NFS USE
744	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
745	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
746
747	# status=$?
748	# IGCM_sys_Rsync_out $status
749
750	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
751	# (( status=status+$? ))
752
753	if [ ${status} -gt 0 ] ; then
754	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
755	[ -f ${1} ] && ls -l ${1}
756	[ -f ${2} ] && ls -l ${2}
757	[ -f ${2}/${1} ] && ls -l ${2}/${1}
758	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
759	IGCM_debug_Exit "IGCM_sys_Put_Out"
760	else
761
762	if [ X${JobType} = XRUN ] ; then
763	if [ X${3} = X ] ; then
764	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
765	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
766	fi
767	fi
768
769	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
770	fi
771	fi
772	IGCM_debug_PopStack "IGCM_sys_Put_Out"
773	return 0
774	}
775
776	#D-#==================================================
777	#D-function IGCM_sys_Get
778	#D-* Purpose: Get a file from ${ARCHIVE}
779	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
780	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
781	function IGCM_sys_Get {
782	IGCM_debug_PushStack "IGCM_sys_Get" $@
783
784	typeset DEST dm_liste target file_work
785	typeset NB_ESSAI DELAI status i
786
787	if ( $DEBUG_sys ) ; then
788	echo "IGCM_sys_Get :" $@
789	fi
790
791	# number of tentative
792	NB_ESSAI=3
793	# time delay between tentative
794	DELAI=2
795
796	if [ $DRYRUN -le 2 ]; then
797	if [ X${1} = X'/l' ] ; then
798	eval set +A dm_liste \${${2}}
799	else
800	eval set +A dm_liste ${1}
801	fi
802	eval DEST=\${${#}}
803	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
804	status=$?
805	if [ ${status} -gt 0 ] ; then
806	echo "WARNING IGCM_sys_Get : error code ${status}"
807	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
808	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
809	fi
810
811	#if [ ${status} -gt 0 ] ; then
812	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
813	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
814	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
815	# sleep 30
816	# echo "We try another time"
817	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
818	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
819	# status=$?
820	# if [ ${status} -gt 0 ] ; then
821	# echo "ERROR IGCM_sys_Get : again demigration error :"
822	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
823	# IGCM_debug_Exit "IGCM_sys_Get"
824	# fi
825	# else
826	# echo "ERROR IGCM_sys_Get : demigration error :"
827	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
828	# IGCM_debug_Exit "IGCM_sys_Get"
829	# fi
830	#fi
831
832	# #RSYNC WITH NETWORK SSH CALL
833	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
834	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
835
836	# #RSYNC WITH NFS USE
837	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
838	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
839
840	# status=$?
841	# IGCM_sys_Rsync_out $status
842
843	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
844	# (( status=status+$? ))
845
846	#USUAL WAY
847	if [ X${1} = X'/l' ] ; then
848	for target in ${dm_liste[*]} ; do
849	local_file=$( basename ${target} )
850	# test if the target file is present before the loop
851	IGCM_sys_TestFileArchive ${target}
852	status=$?
853	if [ ${status} -gt 0 ] ; then
854	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
855	IGCM_debug_Exit "IGCM_sys_Get"
856	else
857	i=0
858	while [ $i -lt $NB_ESSAI ] ; do
859	#if [ X${DoLink} = Xtrue ] ; then
860	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
861	# status=$?
862	# else
863	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
864	# status=$?
865	#fi
866	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
867	status=$?
868	if [ ${status} -gt 0 ]; then
869	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
870	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
871	sleep $DELAI
872	else
873	break
874	fi
875	(( i = i + 1 ))
876	done
877	if [ ${status} -gt 0 ] ; then
878	echo "IGCM_sys_Get : error"
879	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
880	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
881	IGCM_debug_Exit "IGCM_sys_Get"
882	else
883	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
884	fi
885	fi
886	done
887	else
888	i=0
889	while [ $i -lt $NB_ESSAI ] ; do
890	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
891	status=$?
892	if [ ${status} -gt 0 ]; then
893	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
894	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
895	sleep $DELAI
896	else
897	break
898	fi
899	(( i = i + 1 ))
900	done
901	if [ ${status} -gt 0 ] ; then
902	echo "IGCM_sys_Get : error"
903	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
904	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
905	IGCM_debug_Exit "IGCM_sys_Get"
906	else
907	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
908	fi
909	fi
910	fi
911	IGCM_debug_PopStack "IGCM_sys_Get"
912	}
913
914	#D-#==================================================
915	#D-function IGCM_sys_GetDate_Monitoring
916	#D-* Purpose: get the last year for which the monitoring has been computed
917	#D-* Examples:
918	#D-
919	function IGCM_sys_GetDate_Monitoring {
920	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
921	if ( $DEBUG_sys ) ; then
922	echo "IGCM_sys_GetDate_Monitoring :" $@
923	fi
924
925	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
926
927	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
928	}
929
930	#D-#==================================================
931	#D-function IGCM_sys_Dods_Rm
932	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
933	#D-* Examples:
934	#D-
935	function IGCM_sys_Dods_Rm {
936	if ( $DEBUG_sys ) ; then
937	echo "IGCM_sys_Dods_Rm :" $@
938	fi
939	typeset status
940	if [ $DRYRUN = 0 ]; then
941
942	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
943	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
944	# echo "Nothing has been done."
945	# return
946	# fi
947
948	if [ "$#" -eq 1 ]; then
949	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
950	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
951	status=$?
952	else
953	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
954	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
955	status=$?
956	fi
957
958	# if [ ${status} -gt 0 ] ; then
959	# echo "IGCM_sys_Dods_Rm : error."
960	# cat out_dods_rm
961	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
962	# else
963	# rm out_dods_rm
964	# fi
965
966	fi
967	return $status
968	}
969
970	#D-#==================================================
971	#D-function IGCM_sys_Dods_Cp
972	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
973	#D-* Examples:
974	#D-
975	function IGCM_sys_Dods_Cp {
976	if ( $DEBUG_sys ) ; then
977	echo "IGCM_sys_Dods_Cp :" $@
978	fi
979	typeset status
980	if [ $DRYRUN = 0 ]; then
981
982	# if [ ! -d ${R_SAVE}/${1} ] ; then
983	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
984	# echo "Nothing has been done."
985	# return
986	# fi
987
988	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
989	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
990	status=$?
991
992	# if [ ${status} -gt 0 ] ; then
993	# echo "IGCM_sys_Dods_Cp : error."
994	# cat out_dods_cp
995	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
996	# else
997	# rm out_dods_cp
998	# fi
999
1000	fi
1001	return $status
1002	}
1003
1004	#D-#==================================================
1005	#D-function IGCM_sys_Put_Dods
1006	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1007	#D-* Examples:
1008	#D-
1009	function IGCM_sys_Put_Dods {
1010	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1011	if ( $DEBUG_sys ) ; then
1012	echo "IGCM_sys_Put_Dods :" $@
1013	fi
1014	typeset status
1015	if [ $DRYRUN = 0 ]; then
1016	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1017	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1018	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1019	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1020	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1021	return
1022	fi
1023
1024	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1025	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1026	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1027	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1028	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1029	return
1030	fi
1031	#
1032	if [ -d ${R_SAVE}/${1} ] ; then
1033	cd ${R_SAVE}
1034	elif [ -d ${R_FIGR}/${1} ] ; then
1035	cd ${R_FIGR}
1036	fi
1037
1038	IGCM_sys_Dods_Rm ${1}
1039	IGCM_sys_Dods_Cp ${1}
1040	status=0
1041
1042	if [ ${status} -gt 0 ] ; then
1043	echo "IGCM_sys_Put_Dods : error."
1044	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1045	fi
1046	fi
1047	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1048	}
1049
1050	##############################################################
1051	# REBUILD OPERATOR
1052
1053	#D-#==================================================
1054	#D-function IGCM_sys_sync
1055	#D-* Purpose: flush buffer on disk
1056	#D-* Examples:
1057	#D-
1058	function IGCM_sys_sync {
1059	IGCM_debug_PushStack "IGCM_sys_sync" $@
1060	if ( $DEBUG_sys ) ; then
1061	echo "IGCM_sys_sync :" $@
1062	fi
1063
1064	/bin/sync
1065
1066	IGCM_debug_PopStack "IGCM_sys_sync"
1067	}
1068
1069	############################################################
1070	# Activate Running Environnment Variables
1071
1072	#D-#==================================================
1073	#D-function IGCM_sys_activ_variables
1074	#D-* Purpose: set environement variables prior to execution
1075	#D-* Examples:
1076	#D-
1077	function IGCM_sys_activ_variables {
1078	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1079	if ( $DEBUG_sys ) ; then
1080	echo "IGCM_sys_activ_variables"
1081	fi
1082
1083	# --------------------------------------------------------------------
1084	#D- MODULE specifications
1085	# --------------------------------------------------------------------
1086	# Source the file EnvFile containing all module load needed to run the model.
1087	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1088	# it will be used. Otherwise a default file will be used.
1089	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1090	# EnvFile is set in config.card and the file exists
1091	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1092	EnvFile=${config_UserChoices_EnvFile}
1093	else
1094	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1095	module purge > /dev/null 2>&1
1096	if [ X${compilerVersion} = XV17 ] ; then
1097	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
1098	## EnvFile=/ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1099	EnvFile=$( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1100	else
1101	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
1102	fi
1103	fi
1104	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1105	IGCM_debug_Print 1 ${EnvFile}
1106	. ${EnvFile}
1107
1108	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Irene. "
1109	module list
1110
1111	# --------------------------------------------------------------------
1112	#D- MPI specifications
1113	# --------------------------------------------------------------------
1114
1115	# --------------------------------------------------------------------
1116	#D- Other specifications
1117	# --------------------------------------------------------------------
1118
1119	ulimit -s unlimited
1120
1121	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1122	}
1123
1124	############################################################
1125	# Desactivate Running Environnment Variables
1126
1127	#D-#==================================================
1128	#D-function IGCM_sys_desactiv_variables
1129	#D-* Purpose: unset environement variables after execution
1130	#D-* Examples:
1131	#D-
1132	function IGCM_sys_desactiv_variables {
1133	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1134	if ( $DEBUG_sys ) ; then
1135	echo "IGCM_sys_desactiv_variables"
1136	fi
1137	# --------------------------------------------------------------------
1138	#D- MPI specifications
1139	# --------------------------------------------------------------------
1140
1141	# --------------------------------------------------------------------
1142	#D- Other specifications
1143	# --------------------------------------------------------------------
1144
1145	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1146	}
1147
1148	############################################################
1149	# Update job headers to be used by the scheduler
1150
1151	#D-#==================================================
1152	#D-function IGCM_sys_updateHeaders
1153	#D-* Purpose: Update job headers to be used by the scheduler
1154	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1155	#D-
1156	function IGCM_sys_updateHeaders {
1157	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1158	if ( $DEBUG_sys ) ; then
1159	echo "IGCM_sys_updateHeaders"
1160	fi
1161	typeset file
1162	file=$1
1163
1164	if [ ${executionType} -eq 1 ] ; then
1165	# MPMD + MPI
1166	sed -e "/::openMPthreads::/d" \
1167	-e "s/::JobNumProcTot::/${coreNumber}/" \
1168	-e "/#MSUB -x/d" \
1169	-e "/--cpu_bind=none/d" \
1170	${file} > ${file}.tmp
1171
1172	elif [ ${executionType} -eq 2 ] ; then
1173	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1174	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1175	sed -e "/::openMPthreads::/d" \
1176	-e "s/::JobNumProcTot::/${coreNumber}/" \
1177	${file} > ${file}.tmp
1178	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1179	sed -e "/::openMPthreads::/d" \
1180	-e "s/::JobNumProcTot::/${coreNumber}/" \
1181	-e "/--cpu_bind=none/d" \
1182	${file} > ${file}.tmp
1183	else
1184	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1185	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1186	fi
1187	elif [ ${executionType} -eq 3 ] ; then
1188	# SPMD + MPI/OMP
1189	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1190	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1191	-e "/#MSUB -x/d" \
1192	-e "/--cpu_bind=none/d" \
1193	${file} > ${file}.tmp
1194
1195	elif [ ${executionType} -eq 4 ] ; then
1196	# SPMD + MPI only
1197	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1198	-e "/::openMPthreads::/d" \
1199	-e "/#MSUB -x/d" \
1200	-e "/--cpu_bind=none/d" \
1201	${file} > ${file}.tmp
1202
1203	elif [ ${executionType} -eq 5 ] ; then
1204	# SPMD + OMP only
1205	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1206	-e "/::JobNumProcTot::/d" \
1207	-e "/#MSUB -x/d" \
1208	-e "/--cpu_bind=none/d" \
1209	${file} > ${file}.tmp
1210
1211	elif [ ${executionType} -eq 6 ] ; then
1212	# SEQUENTIAL THEN
1213	sed -e "s/::JobNumProcTot::/1/" \
1214	-e "/::openMPthreads::/d" \
1215	-e "/#MSUB -x/d" \
1216	-e "/--cpu_bind=none/d" \
1217	${file} > ${file}.tmp
1218
1219	fi
1220
1221	IGCM_sys_Mv ${file}.tmp ${file}
1222
1223	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1224	}
1225
1226	############################################################
1227	# Build MPI/OMP scripts run file (dummy function)
1228
1229	#D-#==================================================
1230	#D-function IGCM_sys_build_run_file
1231	#D-* Purpose: build run file (deprecated)
1232	#D-* Examples:
1233	#D-
1234	function IGCM_sys_build_run_file {
1235	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1236	}
1237
1238	############################################################
1239	# Build MPI/OMP scripts
1240
1241	#D-#==================================================
1242	#D-function IGCM_sys_build_execution_scripts
1243	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1244	#D-* Examples:
1245	#D-
1246	function IGCM_sys_build_execution_scripts
1247	{
1248	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1249	if ( $DEBUG_sys ) ; then
1250	echo "IGCM_sys_build_execution_scripts " $@
1251	fi
1252
1253	EXECUTION=${HOST_MPIRUN_COMMAND}
1254
1255	if ( ${OK_PARA_MPMD} ) ; then
1256
1257	# MPMD mode
1258	# 1 MPI only
1259	# 2 MPI/OpenMP mpirun method
1260	# 3 MPI/OpenMP ccc_mprun method
1261
1262	if [ -f run_file ] ; then
1263	IGCM_sys_Rm -f run_file
1264	fi
1265	touch run_file
1266
1267	# case 1 : Only MPI (MPMD)
1268	if ( ! ${OK_PARA_OMP} ) ; then
1269
1270	# Build run_file
1271
1272	# First loop on the components for the coupler ie oasis (only if oasis3)
1273	# the coupler ie oasis3 must be the first one
1274	for comp in ${config_ListOfComponents[*]} ; do
1275
1276	eval ExeNameIn=\${config_Executable_${comp}[0]}
1277	eval ExeNameOut=\${config_Executable_${comp}[1]}
1278
1279	# for CPL component only
1280	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1281
1282	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1283	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1284	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1285	fi
1286	done
1287
1288	# Then second loop on the components
1289	for comp in ${config_ListOfComponents[*]} ; do
1290
1291	eval ExeNameIn=\${config_Executable_${comp}[0]}
1292	eval ExeNameOut=\${config_Executable_${comp}[1]}
1293
1294	# Only if we really have an executable for the component and not the coupler ie oasis:
1295	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1296
1297	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1298	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1299	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1300	fi
1301	done
1302
1303	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1304
1305	IGCM_sys_Chmod u+x run_file
1306	if ( $DEBUG_sys ) ; then
1307	echo "run_file contains : "
1308	cat run_file
1309	fi
1310
1311	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1312
1313	# 2 MPI/OpenMP mpirun method
1314
1315	# Use of mpirun instead of ccc_mprun
1316	EXECUTION="time mpirun"
1317
1318	# Hosts treatment
1319	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1320
1321	i=0
1322	rm -f hosts rankfile
1323	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1324	for nodes in `cat hosts.tmp`
1325	do
1326	host[$i]=$nodes
1327	echo "${host[$i]}" >> hosts
1328	IGCM_debug_Print 1 ${host[$i]}
1329	i=$((i+1))
1330	done
1331	rm -f hosts.tmp
1332
1333	listnodes=${host[*]}
1334
1335	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1336
1337	# Initialisation
1338	rank=0
1339	current_core=0
1340	init_exec=n
1341
1342	# Loop on the components
1343	for comp in ${config_ListOfComponents[*]} ; do
1344
1345	eval ExeNameIn=\${config_Executable_${comp}[0]}
1346	eval ExeNameOut=\${config_Executable_${comp}[1]}
1347
1348	# Not possible if oasis has an executable (i.e old version of oasis3)
1349	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1350	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1351	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1352	IGCM_debug_Verif_Exit
1353	fi
1354
1355	# Only if we really have an executable for the component :
1356	if [ "X${ExeNameOut}" != X\"\" ] ; then
1357
1358	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1359	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1360	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1361
1362	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1363	echo "" >> script_${ExeNameOut}.ksh
1364	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1365
1366	# Check if the number of threads is correct
1367	case ${comp_proc_omp_loc} in
1368	2\|4\|8\|16)
1369	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1370	;;
1371	*)
1372	IGCM_debug_Exit "ERROR with OMP parameters !"
1373	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1374	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1375	IGCM_debug_Verif_Exit
1376	;;
1377	esac
1378	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1379	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1380	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1381	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1382	fi
1383
1384	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1385	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1386	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1387	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1388	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1389
1390	if [ ${init_exec} = y ] ; then
1391	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1392	else
1393	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1394	init_exec=y
1395	fi
1396
1397	# Build rankfile : method used to assign cores and nodes for the MPI process
1398	# Ex :
1399	#rank 0=curie5296 slot=0,1,2,3
1400	#rank 1=curie5296 slot=4,5,6,7
1401	# Example of final command :
1402	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1403	# with script_lmdz.x.ksh :
1404	# #!/bin/ksh
1405	#export KMP_STACKSIZE=3g
1406	#export KMP_LIBRARY=turnaround
1407	#export MKL_SERIAL=YES
1408	#OMP_NUM_THREADS=4
1409	#./lmdz.x
1410	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1411	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1412	else
1413	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1414	fi
1415	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1416	(( index_host = current_core / NB_CORE_PER_NODE ))
1417	host_value=${host[${index_host}]}
1418	(( slot = current_core % NB_CORE_PER_NODE ))
1419	virg=","
1420	string_final=""
1421	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1422	string=$index$virg
1423	string_final=$string_final$string
1424	done
1425	string_final=$( echo $string_final \| sed "s/.$//" )
1426	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1427	(( rank = rank + 1 ))
1428	(( current_core = current_core + offset_comp_proc_loc ))
1429	done
1430	fi
1431	done
1432
1433	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1434
1435	# 3 MPI/OpenMP ccc_mprun method
1436
1437	# MPI-OpenMP (MPMD)
1438	# example of run_file
1439	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1440	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1441	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1442
1443	# Loop on the components
1444	for comp in ${config_ListOfComponents[*]} ; do
1445
1446	eval ExeNameIn=\${config_Executable_${comp}[0]}
1447	eval ExeNameOut=\${config_Executable_${comp}[1]}
1448
1449	# Not possible if oasis has an executable (i.e old version of oasis3)
1450	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1451	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1452	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1453	IGCM_debug_Verif_Exit
1454	fi
1455
1456	# Only if we really have an executable for the component :
1457	if [ "X${ExeNameOut}" != X\"\" ] ; then
1458
1459	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1460	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1461
1462	# Build script files
1463
1464	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1465	echo "" >> script_${ExeNameOut}.ksh
1466	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1467
1468	# Check if the number of threads is correct
1469	case ${comp_proc_omp_loc} in
1470	2\|4\|8\|12\|16\|24)
1471	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1472	;;
1473	*)
1474	IGCM_debug_Exit "ERROR with OMP parameters !"
1475	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1476	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1477	IGCM_debug_Verif_Exit
1478	;;
1479	esac
1480	### only for LMDZ?
1481	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1482	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1483	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1484	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1485
1486	fi
1487
1488	# to have out/err per process on different files
1489	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1490	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1491
1492	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1493
1494	# Complete run_file
1495
1496	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1497
1498	fi
1499	done
1500
1501	## variable added to stop after 60s instead of 600s by default.
1502	## This is used when no error comes from executables and when something stopped an executable without notice.
1503	export SLURM_WAIT=60
1504
1505	## module advised by TGCC (instead of 2 variables)
1506	module load feature/bridge/heterogenous_mpmd
1507
1508	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1509
1510	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1511
1512	IGCM_sys_Chmod u+x run_file
1513	if ( $DEBUG_sys ) ; then
1514	echo "run_file contains : "
1515	cat run_file
1516	fi
1517
1518	else
1519
1520	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1521
1522	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1523
1524	else
1525	# Only one executable (SPMD mode).
1526
1527	for comp in ${config_ListOfComponents[*]} ; do
1528
1529	# Only if we really have an executable for the component :
1530	eval ExeNameOut=\${config_Executable_${comp}[1]}
1531	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1532
1533	# Build script files
1534
1535	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1536	echo "" >> script_${ExeNameOut}.ksh
1537	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1538
1539	if ( ${OK_PARA_OMP} ) ; then
1540	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1541	echo "" >> script_${ExeNameOut}.ksh
1542	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1543	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1544	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1545	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1546	fi
1547
1548	if ( ${OK_PARA_MPI} ) ; then
1549	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1550	# Default : ccc_mprun used if nb_proc gt 1
1551	# to have out/err per process on different files
1552	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1553	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1554	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1555	else
1556	# Default : ccc_mprun is NOT used if nb_proc eq 1
1557	# to have out/err per process on different files
1558	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1559	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1560	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1561	fi
1562
1563	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1564	cat script_${ExeNameOut}.ksh
1565
1566	fi
1567
1568	done
1569
1570	fi # ${OK_PARA_MPMD}
1571
1572	IGCM_debug_Print 1 "sys Irene : execution command is "
1573	IGCM_debug_Print 1 "$EXECUTION"
1574
1575	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1576	}
1577
1578	#D-#==================================================
1579	#D-function IGCM_sys_check_path
1580	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1581	#D-* do not point to an important use directory. Stop immediately in that case.
1582	#D-* Examples:
1583	#D-
1584	function IGCM_sys_check_path {
1585	IGCM_debug_PushStack "IGCM_sys_check_path"
1586	if ( $DEBUG_sys ) ; then
1587	echo "IGCM_sys_check_path"
1588	fi
1589
1590	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1591	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1592	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1593	IGCM_debug_Exit "This will stop the job"
1594	fi
1595	IGCM_debug_PopStack "IGCM_sys_check_path"
1596	}
1597
1598	#D-#==================================================
1599	#D-function IGCM_sys_check_quota
1600	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1601	#D-* Examples:
1602	#D-
1603	function IGCM_sys_check_quota {
1604	IGCM_debug_PushStack "IGCM_sys_check_quota"
1605	if ( $DEBUG_sys ) ; then
1606	echo "IGCM_sys_check_quota"
1607	fi
1608	# Limit of quota (in %)
1609	limit_quota=90
1610
1611	# Check of the volume
1612	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1613	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1614
1615	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1616
1617	unit_avail=${volume_avail: -1}
1618	unit_quota=${volume_quota: -1}
1619
1620	if [ "${unit_quota}" = "*" ] ; then
1621	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1622	IGCM_debug_Print 1 "More than 100% of your quota is used"
1623	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1624	IGCM_debug_Print 1 "You must have more than 10% available to run"
1625	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1626	IGCM_debug_Verif_Exit
1627	fi
1628
1629	temp_avail=${volume_avail%%${volume_avail: -1}*}
1630	temp_quota=${volume_quota%%${volume_quota: -1}*}
1631
1632	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1633
1634	# Convertion
1635	if [ ${volume_avail: -1} = "T" ] ; then
1636	(( temp_avail = temp_avail * 1000000000000 ))
1637	elif [ ${volume_avail: -1} = "G" ] ; then
1638	(( temp_avail = temp_avail * 1000000000 ))
1639	elif [ ${volume_avail: -1} = "M" ] ; then
1640	(( temp_avail = temp_avail * 1000000 ))
1641	elif [ ${volume_avail: -1} = "k" ] ; then
1642	(( temp_avail = temp_avail * 1000 ))
1643	else
1644	(( temp_avail = volume_avail ))
1645	fi
1646	if [ ${volume_quota: -1} = "T" ] ; then
1647	(( temp_quota = temp_quota * 1000000000000 ))
1648	elif [ ${volume_quota: -1} = "G" ] ; then
1649	(( temp_quota = temp_quota * 1000000000 ))
1650	elif [ ${volume_quota: -1} = "M" ] ; then
1651	(( temp_quota = temp_quota * 1000000 ))
1652	elif [ ${volume_quota: -1} = "k" ] ; then
1653	(( temp_quota = temp_quota * 1000 ))
1654	else
1655	(( temp_quota = volume_quota ))
1656	fi
1657	fi
1658
1659	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1660	# echo "volume ratio is " $quota_volume
1661
1662	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1663	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1664	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1665	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1666	IGCM_debug_Print 1 "You must have more than 10% available to run"
1667	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1668	IGCM_debug_Verif_Exit
1669	fi
1670
1671	fi
1672
1673	# Check of the number of inodes
1674
1675	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1676	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1677
1678	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1679
1680	unit_avail=${inode_avail: -1}
1681	unit_quota=${inode_quota: -1}
1682
1683	if [ "${unit_quota}" = "*" ] ; then
1684	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1685	IGCM_debug_Print 1 "More than 100% of your quota is used"
1686	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1687	IGCM_debug_Print 1 "You must have more than 10% available to run"
1688	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1689	IGCM_debug_Verif_Exit
1690	fi
1691
1692	temp_avail=${inode_avail%%${inode_avail: -1}*}
1693	temp_quota=${inode_quota%%${inode_quota: -1}*}
1694
1695	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1696
1697	# Convertion
1698	if [ ${inode_avail: -1} = "T" ] ; then
1699	(( temp_avail = temp_avail * 1000000000000 ))
1700	elif [ ${inode_avail: -1} = "G" ] ; then
1701	(( temp_avail = temp_avail * 1000000000 ))
1702	elif [ ${inode_avail: -1} = "M" ] ; then
1703	(( temp_avail = temp_avail * 1000000 ))
1704	elif [ ${inode_avail: -1} = "k" ] ; then
1705	(( temp_avail = temp_avail * 1000 ))
1706	else
1707	(( temp_avail = inode_avail ))
1708	fi
1709
1710	if [ ${inode_quota: -1} = "T" ] ; then
1711	(( temp_quota = temp_quota * 1000000000000 ))
1712	elif [ ${inode_quota: -1} = "G" ] ; then
1713	(( temp_quota = temp_quota * 1000000000 ))
1714	elif [ ${inode_quota: -1} = "M" ] ; then
1715	(( temp_quota = temp_quota * 1000000 ))
1716	elif [ ${inode_quota: -1} = "k" ] ; then
1717	(( temp_quota = temp_quota * 1000 ))
1718	else
1719	(( temp_quota = inode_quota ))
1720	fi
1721	fi
1722	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1723	# echo "inode ratio is " $quota_inode
1724
1725	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1726	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1727	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1728	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1729	IGCM_debug_Print 1 "You must have more than 10% available to run"
1730	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1731	IGCM_debug_Verif_Exit
1732	fi
1733	fi
1734	IGCM_debug_PopStack "IGCM_sys_check_quota"
1735	}
1736
1737	#D-#==================================================
1738	#D-function IGCM_sys_projectAccounting
1739	#D-* Purpose: store project accounting information in a file
1740	#D-* Examples:
1741	#D-
1742	function IGCM_sys_projectAccounting {
1743	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1744	if ( $DEBUG_sys ) ; then
1745	echo "IGCM_sys_check_quota"
1746	fi
1747
1748	ssh irene191 /usr/bin/ccc_myproject > $1
1749
1750	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1751	}
1752
1753	#D-#==================================================
1754	#D-function IGCM_sys_getJobSchedulerID
1755	#D-* Purpose: Get the job ID during execution
1756	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1757	#D-
1758	function IGCM_sys_getJobSchedulerID {
1759	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1760	if ( $DEBUG_sys ) ; then
1761	echo "IGCM_sys_getJobSchedulerID"
1762	fi
1763
1764	eval ${1}=${BRIDGE_MSUB_JOBID}
1765
1766	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1767	}
1768
1769	#D-#==================================================
1770	#D-function IGCM_sys_GetJobID
1771	#D-* Purpose: Get the job ID from the JobName
1772	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1773	#D-
1774	function IGCM_sys_GetJobID {
1775	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1776	if ( $DEBUG_sys ) ; then
1777	echo "IGCM_sys_GetJobID"
1778	fi
1779
1780	# With -f option, the full job name is given in the last column
1781	ID=$( ccc_mstat -f -u $2 \| \
1782	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1783
1784	eval ${3}=${ID}
1785	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1786	}
1787
1788	#D-#==================================================
1789	#D-function IGCM_sys_CountJobInQueue
1790	#D-* Purpose: Count number of users job
1791	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1792	#D-
1793	function IGCM_sys_CountJobInQueue {
1794	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1795	if ( $DEBUG_sys ) ; then
1796	echo "IGCM_sys_CountJobInQueue"
1797	fi
1798
1799	# With -f option, the full job name is given in the last column
1800	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1801
1802	eval ${2}=${NbRun}
1803
1804	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1805	}
1806
1807	#D-#==================================================
1808	#D-function IGCM_sys_ListJobInQueue
1809	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1810	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1811	#D-
1812	function IGCM_sys_ListJobInQueue {
1813	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1814	if ( $DEBUG_sys ) ; then
1815	echo "IGCM_sys_ListJobInQueue"
1816	fi
1817
1818	# With -f option, the full job name is given in the last column
1819	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1820	'( $2 == User && \
1821	$NF != /TS/ && \
1822	$NF !~ /PACK/ && \
1823	$NF !~ /REBUILD/ && \
1824	$NF !~ /pack/ ) \
1825	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1826
1827	eval set -A ${2} ${JobList[*]}
1828
1829	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1830	}
1831
1832	#D-#==================================================
1833	#D-function IGCM_sys_atlas
1834	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1835	#D-* Examples:
1836	#D-
1837	function IGCM_sys_atlas {
1838	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1839	if ( $DEBUG_sys ) ; then
1840	echo "IGCM_sys_atlas :" $@
1841	fi
1842
1843	typeset status
1844
1845	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1846	status=$?
1847	if [ ${status} -gt 0 ] ; then
1848	echo "IGCM_sys_atlas : error code ${status}"
1849	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1850	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1851	IGCM_debug_PopStack "IGCM_sys_atlas"
1852	return 1
1853	else
1854	IGCM_debug_PopStack "IGCM_sys_atlas"
1855	return 0
1856	fi
1857
1858	IGCM_debug_PopStack "IGCM_sys_atlas"
1859	}
1860
1861	#D-#==================================================
1862	#D-function IGCM_sys_rebuild_nemo
1863	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1864	#D-* Examples:
1865	#D-
1866
1867	function IGCM_sys_rebuild_nemo {
1868	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1869	if ( $DEBUG_sys ) ; then
1870	echo "IGCM_sys_rebuild_nemo :" $@
1871	fi
1872
1873	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1874
1875	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1876	}
1877

Note: See TracBrowser for help on using the repository browser.

Download in other formats: