Context Navigation

source: branches/libIGCM_CESMEP/libIGCM_sys/libIGCM_sys_irene.ksh @ 1567

Last change on this file since 1567 was 1567, checked in by ssenesi, 17 months ago
house-keeping scripts hanlde CESMEP outputs
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 63.8 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Irene
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=irene
71	# add default project on irene
72	if [ ! -z $BRIDGE_MSUB_PROJECT ]; then
73	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT} \| cut -d@ -f1 )
74	else
75	typeset PROJECT=$(echo ${CCCHOME} \| cut -d/ -f5 )
76	fi
77	# jobWarningDelay in seconds
78	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
79
80	#D-
81	#D-#==================================================
82	#D-Program used in libIGCM
83	#D-#==================================================
84
85	# Submit command
86	typeset SUBMIT=${SUBMIT:=ccc_msub}
87	# rsync with path
88	typeset -r RSYNC=/usr/bin/rsync
89	# RSYNC_opt args to rsync
90	typeset -r RSYNC_opt="-va"
91	# ie storage filesystem
92	typeset -r STOREHOST=${MASTER}
93	typeset -r REMOTE_RSYNC=/usr/bin/rsync
94
95	#====================================================
96	# Set environment tools (ferret, nco, cdo, rebuild, ...)
97	#====================================================
98	if [ X${TaskType} = Xcomputing ] ; then
99	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
100	else
101	module purge > /dev/null 2>&1
102	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
103	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
104	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
105	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
106
107	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
108	export UVCDAT_ANONYMOUS_LOG=FALSE
109	fi
110
111	# FYI
112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
113	[ ! X${TaskType} = Xchecking ] && module list
114
115	# For AddNoise and AddPertu3DOCE
116	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
117	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
118
119	# For rebuild
120	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
121	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
122
123	#====================================================
124	# Host specific DIRECTORIES
125	#====================================================
126
127	# ============ CESIUM START ============ #
128
129	#====================================================
130	#- Mirror libIGCM from titane to cesium if needed
131	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
132	#if [ ! ${ROOTSYS} = "home" ] ; then
133	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
134	#else
135	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
136	#fi
137
138	#====================================================
139	#- libIGCM_POST
140	#if ( ${MirrorlibIGCM} ) ; then
141	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
142	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
143	#else
144	# typeset -r libIGCM_POST=${libIGCM}
145	#fi
146
147	# ============ CESIUM END ============ #
148
149	#====================================================
150	#- MirrorlibIGCM for frontend
151	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
152
153	#====================================================
154	#- libIGCM_POST for frontend
155	typeset -r libIGCM_POST=${libIGCM}
156
157	#====================================================
158	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
159	typeset -r R_EXE="${MODIPSL}/bin"
160
161	#====================================================
162	#- SUBMIT_DIR : submission dir
163	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
164
165	#====================================================
166	#- IN
167	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
168
169	#====================================================
170	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
171	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
172
173	#====================================================
174	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
175	typeset -r OUTCOMMAND_PATH=/tmp
176
177	#====================================================
178	#- HOST_MPIRUN_COMMAND
179	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
180
181	#====================================================
182	#- Max number of arguments passed to nco operator or demigration command
183	UNIX_MAX_LIMIT=360
184
185	#====================================================
186	#- set PackDefault to true on irene
187	PackDefault=true
188
189	#====================================================
190	#- Default number of MPI task for IPSL coupled model
191	#- required for backward compatibility
192	#-
193	DEFAULT_NUM_PROC_OCE=5
194	DEFAULT_NUM_PROC_CPL=1
195	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
196
197	#====================================================
198	#- Number of computing cores per node
199	#-
200	NB_CORE_PER_NODE=48
201
202	#D-#==================================================
203	#D-function IGCM_sys_defineArchives
204	#D-* Purpose:
205	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
206	#D-* Define ARCHIVE : Dedicated to large files
207	#D-* Define STORAGE : Dedicated to small/medium files
208	#D-* Define R_OUT : Output tree located on ARCHIVE
209	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
210	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
211	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
212	#D-* Examples:
213	#D-
214	function IGCM_sys_defineArchives {
215	IGCM_debug_PushStack "IGCM_sys_defineArchives"
216	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
217	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = XDEFAULT ]; then
218	# Default option: Change the dfldatadir according to the project used for submission
219	# The variable DataProject in section UserChoices in config.card is not set or is set to DEFAULT
220	if [ X"$( echo ${PROJECT} \| grep cmip6 )" != "X" ] ; then
221	module switch dfldatadir dfldatadir/gencmip6
222	IGCM_debug_Print 1 "Load dfldatadir for forced gencmip6 project"
223	else
224	echo "Load dfldatadir for project ${PROJECT}"
225	module switch dfldatadir dfldatadir/${PROJECT}
226	IGCM_debug_Print 1 "Load dfldatadir for project ${PROJECT}"
227	fi
228	else
229	# Use the project set by the variable DataProject in section UserChoices in config.card
230	module switch dfldatadir dfldatadir/${config_UserChoices_DataProject}
231	IGCM_debug_Print 1 "Load dfldatadir for DataProject read in config.card : ${config_UserChoices_DataProject}"
232	fi
233	#====================================================
234	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
235	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
236
237	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
238	#====================================================
239	#- ARCHIVE (dedicated to large files)
240	ARCHIVE=${config_UserChoices_ARCHIVE}
241	else
242	#====================================================
243	#- ARCHIVE (dedicated to large files)
244	ARCHIVE=${CCCSTOREDIR}
245	fi
246
247	if [ ! X${config_UserChoices_STORAGE} = X ]; then
248	#====================================================
249	#- STORAGE (dedicated to small/medium files)
250	STORAGE=${config_UserChoices_STORAGE}
251	else
252	#====================================================
253	#- STORAGE (dedicated to small/medium files)
254	STORAGE=${CCCWORKDIR}
255	fi
256
257	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
258	#====================================================
259	#- R_OUT
260	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
261
262	#====================================================
263	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
264	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
265
266	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
267
268	else
269	#====================================================
270	#- R_OUT
271	R_OUT=${ARCHIVE}/IGCM_OUT
272
273	#====================================================
274	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
275	R_FIG=${STORAGE}/IGCM_OUT
276	fi
277
278	#====================================================
279	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
280	CMIP6_BUF=${STORAGE}/IGCM_OUT
281
282	#====================================================
283	#- R_BUF (ONLY FOR double copy an scratch)
284	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
285
286	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
287	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
288	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
289
290	IGCM_debug_PopStack "IGCM_sys_defineArchives"
291	}
292
293	#D-#==================================================
294	#D-function IGCM_sys_RshArchive
295	#D-* Purpose: Archive rsh command
296	#D-* Examples:
297	#D-
298	function IGCM_sys_RshArchive {
299	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
300	/bin/ksh <<-EOF
301	${@}
302	EOF
303	status=$?
304	if [ ${status} -gt 0 ] ; then
305	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
306	IGCM_debug_Exit "IGCM_sys_RshArchive"
307	fi
308	IGCM_debug_PopStack "IGCM_sys_RshArchive"
309	}
310
311	#D-#==================================================
312	#D-function IGCM_sys_RshArchive_NoError
313	#D-* Purpose: Archive rsh command, without error
314	#D-* used only in monitoring.job
315	#D-* Examples:
316	#D-
317	function IGCM_sys_RshArchive_NoError {
318	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
319	/bin/ksh <<-EOF
320	${@} 2> /dev/null
321	EOF
322	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
323	}
324
325	#D-#==================================================
326	#D-function IGCM_sys_MkdirArchive
327	#D-* Purpose: Mkdir on Archive
328	#D-* Examples:
329	#D-
330	function IGCM_sys_MkdirArchive {
331	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
332	if ( $DEBUG_sys ) ; then
333	echo "IGCM_sys_MkdirArchive :" $@
334	fi
335	#- creation de repertoire sur le serveur fichier
336	if [ ! -d ${1} ]; then
337	\mkdir -p $1
338	status=$?
339
340	if [ ${status} -gt 0 ] ; then
341	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
342	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
343	fi
344	fi
345	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
346	}
347
348	#D-#==================================================
349	#D-function IGCM_sys_TestDirArchive
350	#D-* Purpose: Test Directory that must exists on Archive
351	#D-* Examples:
352	#D-
353	function IGCM_sys_TestDirArchive {
354	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
355	if ( $DEBUG_sys ) ; then
356	echo "IGCM_sys_TestDirArchive :" $@
357	fi
358	typeset ExistFlag
359	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
360	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
361	return ${ExistFlag}
362	}
363
364	#D-#==================================================
365	#D-function IGCM_sys_IsFileArchived
366	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
367	#D-* Examples:
368	#D-
369	function IGCM_sys_IsFileArchived {
370	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
371	if ( $DEBUG_sys ) ; then
372	echo "IGCM_sys_IsFileArchived :" $@
373	fi
374	typeset IsArchivedFlag
375	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
376	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
377
378	return ${IsArchivedFlag}
379	}
380
381	#D-#==================================================
382	#D-function IGCM_sys_TestFileArchive
383	#D-* Purpose: Test file that must NOT EXISTS on Archive
384	#D-* Examples:
385	#D-
386	function IGCM_sys_TestFileArchive {
387	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
388	typeset ExistFlag
389	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
390	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
391
392	return ${ExistFlag}
393	}
394
395	#D-#==================================================
396	#D-function IGCM_sys_CountFileArchive
397	#D-* Purpose: Count files on Archive filesystem
398	#D-* Examples:
399	#D-
400	function IGCM_sys_CountFileArchive {
401	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
402	ls ${@} 2>/dev/null \| wc -l
403	if [ $? -gt 0 ] ; then
404	echo "IGCM_sys_CountFileArchive : erreur."
405	fi
406	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
407	}
408
409	#D-#==================================================
410	#D-function IGCM_sys_Tree
411	#D-* Purpose: Tree directories with files on ${ARCHIVE}
412	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
413	#D-
414	function IGCM_sys_Tree {
415	IGCM_debug_PushStack "IGCM_sys_Tree" $@
416	if ( $DEBUG_sys ) ; then
417	echo "IGCM_sys_Tree :" $@
418	fi
419
420	\ls -lR ${@}
421
422	IGCM_debug_PopStack "IGCM_sys_Tree"
423	}
424
425	#D-#==================================================
426	#D-function IGCM_sys_Qsub
427	#D-* Purpose: Qsub new job
428	#D-* Examples:
429	#D-
430	function IGCM_sys_Qsub {
431	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
432	if ( $DEBUG_sys ) ; then
433	echo "IGCM_sys_Qsub :" $@
434	fi
435	typeset options status
436	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
437
438	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
439	status=$?
440
441	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
442	if [ ${status} -gt 0 ] ; then
443	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
444	IGCM_debug_Exit "IGCM_sys_Qsub"
445	else
446	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
447	fi
448	IGCM_debug_PopStack "IGCM_sys_Qsub"
449	}
450
451	#D-#==================================================
452	#D-function IGCM_sys_QsubPost
453	#D-* Purpose: Qsub new job on scalaire
454	#D-* Examples:
455	#D-
456	function IGCM_sys_QsubPost {
457	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
458	if ( $DEBUG_sys ) ; then
459	echo "IGCM_sys_QsubPost :" $@
460	fi
461	typeset options status
462	options="-Q normal -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
463
464	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
465	status=$?
466
467	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
468	if [ ${status} -gt 0 ] ; then
469	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
470	IGCM_debug_Exit "IGCM_sys_QsubPost"
471	else
472	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
473	fi
474	IGCM_debug_PopStack "IGCM_sys_QsubPost"
475	}
476
477	#D-*************************
478	#D- File transfer functions
479	#D-*************************
480	#D-
481
482	#D-#==================================================
483	#D-function IGCM_sys_RmRunDir
484	#D-* Purpose: rm tmpdir (dummy function most of the time batch
485	#D- scheduler will do the job)
486	#D-* Examples:
487	#D-
488	function IGCM_sys_RmRunDir {
489	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
490	if ( $DEBUG_sys ) ; then
491	echo "IGCM_sys_RmRunDir :" $@
492	fi
493
494	typeset status
495
496	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
497	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
498	status=$?
499
500	if [ ${status} -gt 0 ] ; then
501	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
502	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
503	IGCM_debug_Exit "IGCM_sys_RmRunDir"
504	else
505	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
506	fi
507	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
508	}
509
510	#D-#==================================================
511	#D-function IGCM_sys_Put_Dir
512	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
513	#D-* Examples:
514	#D-
515	function IGCM_sys_Put_Dir {
516	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
517	if ( $DEBUG_sys ) ; then
518	echo "IGCM_sys_Put_Dir :" $@
519	fi
520	if [ $DRYRUN = 0 ]; then
521	if [ ! -d ${1} ] ; then
522	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
523	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
524	return
525	fi
526
527	typeset status
528
529	# Only if we use rsync
530	#IGCM_sys_TestDirArchive $( dirname $2 )
531	#
532	#USUAL WAY
533	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
534	status=$?
535
536	if [ ${status} -gt 0 ] ; then
537	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
538	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
539	IGCM_debug_Exit "IGCM_sys_Put_Dir"
540	else
541	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
542	fi
543	fi
544	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
545	}
546
547	#D-#==================================================
548	#D-function IGCM_sys_Get_Dir
549	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
550	#D-* Examples:
551	#D-
552	function IGCM_sys_Get_Dir {
553	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
554	if ( $DEBUG_sys ) ; then
555	echo "IGCM_sys_Get_Dir :" $@
556	fi
557	if [ $DRYRUN = 0 ]; then
558	typeset NB_ESSAI DELAI status i
559	# number of tentative
560	NB_ESSAI=3
561	# time delay between tentative
562	DELAI=2
563
564	#
565	# USUAL WAY
566	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
567	ccc_hsm get -r $1
568
569	i=0
570	while [ $i -lt $NB_ESSAI ] ; do
571	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
572	status=$?
573	if [ ${status} -gt 0 ] ; then
574	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
575	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
576	sleep $DELAI
577	else
578	break
579	fi
580	(( i = i + 1 ))
581	done
582
583	if [ ${status} -gt 0 ] ; then
584	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
585	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
586	IGCM_debug_Exit "IGCM_sys_Get_Dir"
587	else
588	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
589	fi
590	fi
591	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
592	}
593
594	#D-#==================================================
595	#D-function IGCM_sys_Put_Rest
596	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
597	#D- File and target directory must exist.
598	#D-* Examples:
599	#D-
600	function IGCM_sys_Put_Rest {
601	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
602	if ( $DEBUG_sys ) ; then
603	echo "IGCM_sys_Put_Rest :" $@
604	fi
605	if [ $DRYRUN = 0 ]; then
606	if [ ! -f ${1} ] ; then
607	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
608	IGCM_debug_Exit "IGCM_sys_Put_Rest"
609	fi
610
611	typeset status
612	#
613	# USUAL WAY
614	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
615	status=$?
616
617	# #RSYNC WITH NETWORK SSH CALL
618	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
619	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
620
621	# #RSYNC WITH NFS USE
622	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
623	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
624
625	# status=$?
626	# IGCM_sys_Rsync_out $status
627
628	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
629	# (( status=status+$? ))
630
631	if [ ${status} -gt 0 ] ; then
632	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
633	[ -f ${1} ] && ls -l ${1}
634	[ -f ${2} ] && ls -l ${2}
635	[ -f ${2}/${1} ] && ls -l ${2}/${1}
636	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
637	IGCM_debug_Exit "IGCM_sys_Put_Rest"
638	else
639
640	if [ X${JobType} = XRUN ] ; then
641	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
642	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
643	fi
644
645	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
646	fi
647	fi
648	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
649	}
650
651	#D-#==================================================
652	#D-function IGCM_sys_Put_Out
653	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
654	#D-* Examples:
655	#D-
656	function IGCM_sys_Put_Out {
657	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
658	if ( $DEBUG_sys ) ; then
659	echo "IGCM_sys_Put_Out :" $@
660	fi
661
662	typeset NB_ESSAI DELAI status i exist skip
663	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
664
665	# number of tentative
666	NB_ESSAI=3
667	# time delay between tentative
668	DELAI=2
669
670	if [ $DRYRUN = 0 ]; then
671	if [ ! -f ${1} ] ; then
672	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
673	IGCM_debug_PopStack "IGCM_sys_Put_Out"
674	return 1
675	fi
676	#
677	IGCM_sys_MkdirArchive $( dirname $2 )
678	#
679	exist=false
680	skip=false
681	if [ -f $2 ] ; then
682	IGCM_debug_Print 1 "$2 already exist"
683	ccc_hsm get $2
684	exist=true
685	if [ "X$( diff $1 $2 )" = X ] ; then
686	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
687	skip=true
688	else
689	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
690	skip=false
691	fi
692	fi
693	#
694	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
695	IGCM_sys_Chmod u+w $2
696	fi
697
698	if [ X${skip} = Xfalse ] ; then
699	i=0
700	while [ $i -lt $NB_ESSAI ] ; do
701	# Identify file system
702	fileDeviceNumberInHex=$( stat -c %d $1 )
703	status=$?
704	if [ ${status} -gt 0 ] ; then
705	IGCM_debug_Exit "IGCM_sys_Put_Out"
706	fi
707	# Identify file system
708	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
709	status=$?
710	if [ ${status} -gt 0 ] ; then
711	IGCM_debug_Exit "IGCM_sys_Put_Out"
712	fi
713
714	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
715	# They are not on the same device. USUAL WAY
716	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
717	status=$?
718	else
719	# They are on the same device. NOT SO USUAL WAY
720	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
721	status=$?
722	fi
723	if [ ${status} -gt 0 ]; then
724	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
725	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
726	[ -f ${1} ] && ls -l ${1}
727	[ -f ${2} ] && ls -l ${2}
728	[ -f ${2}/${1} ] && ls -l ${2}/${1}
729	sleep $DELAI
730	else
731	break
732	fi
733	(( i = i + 1 ))
734	done
735	fi
736
737	# #RSYNC WITH NETWORK SSH CALL
738	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
739	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
740
741	# #RSYNC WITH NFS USE
742	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
743	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
744
745	# status=$?
746	# IGCM_sys_Rsync_out $status
747
748	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
749	# (( status=status+$? ))
750
751	if [ ${status} -gt 0 ] ; then
752	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
753	[ -f ${1} ] && ls -l ${1}
754	[ -f ${2} ] && ls -l ${2}
755	[ -f ${2}/${1} ] && ls -l ${2}/${1}
756	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
757	IGCM_debug_Exit "IGCM_sys_Put_Out"
758	else
759
760	if [ X${JobType} = XRUN ] ; then
761	if [ X${3} = X ] ; then
762	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
763	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
764	fi
765	fi
766
767	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
768	fi
769	fi
770	IGCM_debug_PopStack "IGCM_sys_Put_Out"
771	return 0
772	}
773
774	#D-#==================================================
775	#D-function IGCM_sys_Get
776	#D-* Purpose: Get a file from ${ARCHIVE}
777	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
778	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
779	function IGCM_sys_Get {
780	IGCM_debug_PushStack "IGCM_sys_Get" $@
781
782	typeset DEST dm_liste target file_work
783	typeset NB_ESSAI DELAI status i
784
785	if ( $DEBUG_sys ) ; then
786	echo "IGCM_sys_Get :" $@
787	fi
788
789	# number of tentative
790	NB_ESSAI=3
791	# time delay between tentative
792	DELAI=2
793
794	if [ $DRYRUN -le 2 ]; then
795	if [ X${1} = X'/l' ] ; then
796	eval set +A dm_liste \${${2}}
797	else
798	eval set +A dm_liste ${1}
799	fi
800	eval DEST=\${${#}}
801	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
802	status=$?
803	if [ ${status} -gt 0 ] ; then
804	echo "WARNING IGCM_sys_Get : error code ${status}"
805	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
806	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
807	fi
808
809	#if [ ${status} -gt 0 ] ; then
810	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
811	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
812	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
813	# sleep 30
814	# echo "We try another time"
815	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
816	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
817	# status=$?
818	# if [ ${status} -gt 0 ] ; then
819	# echo "ERROR IGCM_sys_Get : again demigration error :"
820	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
821	# IGCM_debug_Exit "IGCM_sys_Get"
822	# fi
823	# else
824	# echo "ERROR IGCM_sys_Get : demigration error :"
825	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
826	# IGCM_debug_Exit "IGCM_sys_Get"
827	# fi
828	#fi
829
830	# #RSYNC WITH NETWORK SSH CALL
831	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
832	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
833
834	# #RSYNC WITH NFS USE
835	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
836	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
837
838	# status=$?
839	# IGCM_sys_Rsync_out $status
840
841	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
842	# (( status=status+$? ))
843
844	#USUAL WAY
845	if [ X${1} = X'/l' ] ; then
846	for target in ${dm_liste[*]} ; do
847	local_file=$( basename ${target} )
848	# test if the target file is present before the loop
849	IGCM_sys_TestFileArchive ${target}
850	status=$?
851	if [ ${status} -gt 0 ] ; then
852	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
853	IGCM_debug_Exit "IGCM_sys_Get"
854	else
855	i=0
856	while [ $i -lt $NB_ESSAI ] ; do
857	#if [ X${DoLink} = Xtrue ] ; then
858	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
859	# status=$?
860	# else
861	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
862	# status=$?
863	#fi
864	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
865	status=$?
866	if [ ${status} -gt 0 ]; then
867	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
868	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
869	sleep $DELAI
870	else
871	break
872	fi
873	(( i = i + 1 ))
874	done
875	if [ ${status} -gt 0 ] ; then
876	echo "IGCM_sys_Get : error"
877	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
878	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
879	IGCM_debug_Exit "IGCM_sys_Get"
880	else
881	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
882	fi
883	fi
884	done
885	else
886	i=0
887	while [ $i -lt $NB_ESSAI ] ; do
888	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
889	status=$?
890	if [ ${status} -gt 0 ]; then
891	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
892	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
893	sleep $DELAI
894	else
895	break
896	fi
897	(( i = i + 1 ))
898	done
899	if [ ${status} -gt 0 ] ; then
900	echo "IGCM_sys_Get : error"
901	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
902	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
903	IGCM_debug_Exit "IGCM_sys_Get"
904	else
905	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
906	fi
907	fi
908	fi
909	IGCM_debug_PopStack "IGCM_sys_Get"
910	}
911
912	#D-#==================================================
913	#D-function IGCM_sys_GetDate_Monitoring
914	#D-* Purpose: get the last year for which the monitoring has been computed
915	#D-* Examples:
916	#D-
917	function IGCM_sys_GetDate_Monitoring {
918	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
919	if ( $DEBUG_sys ) ; then
920	echo "IGCM_sys_GetDate_Monitoring :" $@
921	fi
922
923	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
924
925	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
926	}
927
928	#D-#==================================================
929	#D-function IGCM_sys_Dods_Rm
930	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
931	#D-* Examples:
932	#D-
933	function IGCM_sys_Dods_Rm {
934	if ( $DEBUG_sys ) ; then
935	echo "IGCM_sys_Dods_Rm :" $@
936	fi
937	typeset status
938	if [ $DRYRUN = 0 ]; then
939
940	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
941	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
942	# echo "Nothing has been done."
943	# return
944	# fi
945
946	if [ "$#" -eq 1 ]; then
947	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
948	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
949	status=$?
950	else
951	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
952	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
953	status=$?
954	fi
955
956	# if [ ${status} -gt 0 ] ; then
957	# echo "IGCM_sys_Dods_Rm : error."
958	# cat out_dods_rm
959	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
960	# else
961	# rm out_dods_rm
962	# fi
963
964	fi
965	return $status
966	}
967
968	#D-#==================================================
969	#D-function IGCM_sys_Dods_Cp
970	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
971	#D-* Examples:
972	#D-
973	function IGCM_sys_Dods_Cp {
974	if ( $DEBUG_sys ) ; then
975	echo "IGCM_sys_Dods_Cp :" $@
976	fi
977	typeset status
978	if [ $DRYRUN = 0 ]; then
979
980	# if [ ! -d ${R_SAVE}/${1} ] ; then
981	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
982	# echo "Nothing has been done."
983	# return
984	# fi
985
986	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
987	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
988	status=$?
989
990	# if [ ${status} -gt 0 ] ; then
991	# echo "IGCM_sys_Dods_Cp : error."
992	# cat out_dods_cp
993	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
994	# else
995	# rm out_dods_cp
996	# fi
997
998	fi
999	return $status
1000	}
1001
1002	#D-#==================================================
1003	#D-function IGCM_sys_Put_Dods
1004	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1005	#D-* Examples:
1006	#D-
1007	function IGCM_sys_Put_Dods {
1008	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1009	if ( $DEBUG_sys ) ; then
1010	echo "IGCM_sys_Put_Dods :" $@
1011	fi
1012	typeset status
1013	if [ $DRYRUN = 0 ]; then
1014	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1015	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1016	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1017	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1018	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1019	return
1020	fi
1021
1022	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1023	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1024	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1025	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1026	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1027	return
1028	fi
1029	#
1030	if [ -d ${R_SAVE}/${1} ] ; then
1031	cd ${R_SAVE}
1032	elif [ -d ${R_FIGR}/${1} ] ; then
1033	cd ${R_FIGR}
1034	fi
1035
1036	IGCM_sys_Dods_Rm ${1}
1037	IGCM_sys_Dods_Cp ${1}
1038	status=0
1039
1040	if [ ${status} -gt 0 ] ; then
1041	echo "IGCM_sys_Put_Dods : error."
1042	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1043	fi
1044	fi
1045	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1046	}
1047
1048	##############################################################
1049	# REBUILD OPERATOR
1050
1051	#D-#==================================================
1052	#D-function IGCM_sys_sync
1053	#D-* Purpose: flush buffer on disk
1054	#D-* Examples:
1055	#D-
1056	function IGCM_sys_sync {
1057	IGCM_debug_PushStack "IGCM_sys_sync" $@
1058	if ( $DEBUG_sys ) ; then
1059	echo "IGCM_sys_sync :" $@
1060	fi
1061
1062	/bin/sync
1063
1064	IGCM_debug_PopStack "IGCM_sys_sync"
1065	}
1066
1067	############################################################
1068	# Activate Running Environnment Variables
1069
1070	#D-#==================================================
1071	#D-function IGCM_sys_activ_variables
1072	#D-* Purpose: set environement variables prior to execution
1073	#D-* Examples:
1074	#D-
1075	function IGCM_sys_activ_variables {
1076	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1077	if ( $DEBUG_sys ) ; then
1078	echo "IGCM_sys_activ_variables"
1079	fi
1080
1081	# --------------------------------------------------------------------
1082	#D- MODULE specifications
1083	# --------------------------------------------------------------------
1084	# Source the file EnvFile containing all module load needed to run the model.
1085	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1086	# it will be used. Otherwise a default file will be used.
1087	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1088	# EnvFile is set in config.card and the file exists
1089	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1090	EnvFile=${config_UserChoices_EnvFile}
1091	else
1092	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1093	module purge > /dev/null 2>&1
1094	if [ X${compilerVersion} = XV17 ] ; then
1095	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
1096	## EnvFile=/ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1097	EnvFile=$( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1098	else
1099	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
1100	fi
1101	fi
1102	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1103	IGCM_debug_Print 1 ${EnvFile}
1104	. ${EnvFile}
1105
1106	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Irene. "
1107	module list
1108
1109	# --------------------------------------------------------------------
1110	#D- MPI specifications
1111	# --------------------------------------------------------------------
1112
1113	# --------------------------------------------------------------------
1114	#D- Other specifications
1115	# --------------------------------------------------------------------
1116
1117	ulimit -s unlimited
1118
1119	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1120	}
1121
1122	############################################################
1123	# Desactivate Running Environnment Variables
1124
1125	#D-#==================================================
1126	#D-function IGCM_sys_desactiv_variables
1127	#D-* Purpose: unset environement variables after execution
1128	#D-* Examples:
1129	#D-
1130	function IGCM_sys_desactiv_variables {
1131	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1132	if ( $DEBUG_sys ) ; then
1133	echo "IGCM_sys_desactiv_variables"
1134	fi
1135	# --------------------------------------------------------------------
1136	#D- MPI specifications
1137	# --------------------------------------------------------------------
1138
1139	# --------------------------------------------------------------------
1140	#D- Other specifications
1141	# --------------------------------------------------------------------
1142
1143	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1144	}
1145
1146	############################################################
1147	# Update job headers to be used by the scheduler
1148
1149	#D-#==================================================
1150	#D-function IGCM_sys_updateHeaders
1151	#D-* Purpose: Update job headers to be used by the scheduler
1152	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1153	#D-
1154	function IGCM_sys_updateHeaders {
1155	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1156	if ( $DEBUG_sys ) ; then
1157	echo "IGCM_sys_updateHeaders"
1158	fi
1159	typeset file
1160	file=$1
1161
1162	if [ ${executionType} -eq 1 ] ; then
1163	# MPMD + MPI
1164	sed -e "/::openMPthreads::/d" \
1165	-e "s/::JobNumProcTot::/${coreNumber}/" \
1166	-e "/#MSUB -x/d" \
1167	-e "/--cpu_bind=none/d" \
1168	${file} > ${file}.tmp
1169
1170	elif [ ${executionType} -eq 2 ] ; then
1171	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1172	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1173	sed -e "/::openMPthreads::/d" \
1174	-e "s/::JobNumProcTot::/${coreNumber}/" \
1175	${file} > ${file}.tmp
1176	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1177	sed -e "/::openMPthreads::/d" \
1178	-e "s/::JobNumProcTot::/${coreNumber}/" \
1179	-e "/--cpu_bind=none/d" \
1180	${file} > ${file}.tmp
1181	else
1182	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1183	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1184	fi
1185	elif [ ${executionType} -eq 3 ] ; then
1186	# SPMD + MPI/OMP
1187	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1188	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1189	-e "/#MSUB -x/d" \
1190	-e "/--cpu_bind=none/d" \
1191	${file} > ${file}.tmp
1192
1193	elif [ ${executionType} -eq 4 ] ; then
1194	# SPMD + MPI only
1195	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1196	-e "/::openMPthreads::/d" \
1197	-e "/#MSUB -x/d" \
1198	-e "/--cpu_bind=none/d" \
1199	${file} > ${file}.tmp
1200
1201	elif [ ${executionType} -eq 5 ] ; then
1202	# SPMD + OMP only
1203	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1204	-e "/::JobNumProcTot::/d" \
1205	-e "/#MSUB -x/d" \
1206	-e "/--cpu_bind=none/d" \
1207	${file} > ${file}.tmp
1208
1209	elif [ ${executionType} -eq 6 ] ; then
1210	# SEQUENTIAL THEN
1211	sed -e "s/::JobNumProcTot::/1/" \
1212	-e "/::openMPthreads::/d" \
1213	-e "/#MSUB -x/d" \
1214	-e "/--cpu_bind=none/d" \
1215	${file} > ${file}.tmp
1216
1217	fi
1218
1219	IGCM_sys_Mv ${file}.tmp ${file}
1220
1221	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1222	}
1223
1224	############################################################
1225	# Build MPI/OMP scripts run file (dummy function)
1226
1227	#D-#==================================================
1228	#D-function IGCM_sys_build_run_file
1229	#D-* Purpose: build run file (deprecated)
1230	#D-* Examples:
1231	#D-
1232	function IGCM_sys_build_run_file {
1233	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1234	}
1235
1236	############################################################
1237	# Build MPI/OMP scripts
1238
1239	#D-#==================================================
1240	#D-function IGCM_sys_build_execution_scripts
1241	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1242	#D-* Examples:
1243	#D-
1244	function IGCM_sys_build_execution_scripts
1245	{
1246	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1247	if ( $DEBUG_sys ) ; then
1248	echo "IGCM_sys_build_execution_scripts " $@
1249	fi
1250
1251	EXECUTION=${HOST_MPIRUN_COMMAND}
1252
1253	if ( ${OK_PARA_MPMD} ) ; then
1254
1255	# MPMD mode
1256	# 1 MPI only
1257	# 2 MPI/OpenMP mpirun method
1258	# 3 MPI/OpenMP ccc_mprun method
1259
1260	if [ -f run_file ] ; then
1261	IGCM_sys_Rm -f run_file
1262	fi
1263	touch run_file
1264
1265	# case 1 : Only MPI (MPMD)
1266	if ( ! ${OK_PARA_OMP} ) ; then
1267
1268	# Build run_file
1269
1270	# First loop on the components for the coupler ie oasis (only if oasis3)
1271	# the coupler ie oasis3 must be the first one
1272	for comp in ${config_ListOfComponents[*]} ; do
1273
1274	eval ExeNameIn=\${config_Executable_${comp}[0]}
1275	eval ExeNameOut=\${config_Executable_${comp}[1]}
1276
1277	# for CPL component only
1278	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1279
1280	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1281	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1282	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1283	fi
1284	done
1285
1286	# Then second loop on the components
1287	for comp in ${config_ListOfComponents[*]} ; do
1288
1289	eval ExeNameIn=\${config_Executable_${comp}[0]}
1290	eval ExeNameOut=\${config_Executable_${comp}[1]}
1291
1292	# Only if we really have an executable for the component and not the coupler ie oasis:
1293	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1294
1295	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1296	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1297	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1298	fi
1299	done
1300
1301	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1302
1303	IGCM_sys_Chmod u+x run_file
1304	if ( $DEBUG_sys ) ; then
1305	echo "run_file contains : "
1306	cat run_file
1307	fi
1308
1309	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1310
1311	# 2 MPI/OpenMP mpirun method
1312
1313	# Use of mpirun instead of ccc_mprun
1314	EXECUTION="time mpirun"
1315
1316	# Hosts treatment
1317	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1318
1319	i=0
1320	rm -f hosts rankfile
1321	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1322	for nodes in `cat hosts.tmp`
1323	do
1324	host[$i]=$nodes
1325	echo "${host[$i]}" >> hosts
1326	IGCM_debug_Print 1 ${host[$i]}
1327	i=$((i+1))
1328	done
1329	rm -f hosts.tmp
1330
1331	listnodes=${host[*]}
1332
1333	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1334
1335	# Initialisation
1336	rank=0
1337	current_core=0
1338	init_exec=n
1339
1340	# Loop on the components
1341	for comp in ${config_ListOfComponents[*]} ; do
1342
1343	eval ExeNameIn=\${config_Executable_${comp}[0]}
1344	eval ExeNameOut=\${config_Executable_${comp}[1]}
1345
1346	# Not possible if oasis has an executable (i.e old version of oasis3)
1347	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1348	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1349	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1350	IGCM_debug_Verif_Exit
1351	fi
1352
1353	# Only if we really have an executable for the component :
1354	if [ "X${ExeNameOut}" != X\"\" ] ; then
1355
1356	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1357	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1358	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1359
1360	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1361	echo "" >> script_${ExeNameOut}.ksh
1362	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1363
1364	# Check if the number of threads is correct
1365	case ${comp_proc_omp_loc} in
1366	2\|4\|8\|16)
1367	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1368	;;
1369	*)
1370	IGCM_debug_Exit "ERROR with OMP parameters !"
1371	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1372	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1373	IGCM_debug_Verif_Exit
1374	;;
1375	esac
1376	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1377	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1378	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1379	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1380	fi
1381
1382	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1383	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1384	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1385	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1386	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1387
1388	if [ ${init_exec} = y ] ; then
1389	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1390	else
1391	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1392	init_exec=y
1393	fi
1394
1395	# Build rankfile : method used to assign cores and nodes for the MPI process
1396	# Ex :
1397	#rank 0=curie5296 slot=0,1,2,3
1398	#rank 1=curie5296 slot=4,5,6,7
1399	# Example of final command :
1400	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1401	# with script_lmdz.x.ksh :
1402	# #!/bin/ksh
1403	#export KMP_STACKSIZE=3g
1404	#export KMP_LIBRARY=turnaround
1405	#export MKL_SERIAL=YES
1406	#OMP_NUM_THREADS=4
1407	#./lmdz.x
1408	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1409	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1410	else
1411	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1412	fi
1413	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1414	(( index_host = current_core / NB_CORE_PER_NODE ))
1415	host_value=${host[${index_host}]}
1416	(( slot = current_core % NB_CORE_PER_NODE ))
1417	virg=","
1418	string_final=""
1419	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1420	string=$index$virg
1421	string_final=$string_final$string
1422	done
1423	string_final=$( echo $string_final \| sed "s/.$//" )
1424	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1425	(( rank = rank + 1 ))
1426	(( current_core = current_core + offset_comp_proc_loc ))
1427	done
1428	fi
1429	done
1430
1431	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1432
1433	# 3 MPI/OpenMP ccc_mprun method
1434
1435	# MPI-OpenMP (MPMD)
1436	# example of run_file
1437	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1438	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1439	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1440
1441	# Loop on the components
1442	for comp in ${config_ListOfComponents[*]} ; do
1443
1444	eval ExeNameIn=\${config_Executable_${comp}[0]}
1445	eval ExeNameOut=\${config_Executable_${comp}[1]}
1446
1447	# Not possible if oasis has an executable (i.e old version of oasis3)
1448	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1449	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1450	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1451	IGCM_debug_Verif_Exit
1452	fi
1453
1454	# Only if we really have an executable for the component :
1455	if [ "X${ExeNameOut}" != X\"\" ] ; then
1456
1457	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1458	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1459
1460	# Build script files
1461
1462	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1463	echo "" >> script_${ExeNameOut}.ksh
1464	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1465
1466	# Check if the number of threads is correct
1467	case ${comp_proc_omp_loc} in
1468	2\|4\|8\|12\|16\|24)
1469	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1470	;;
1471	*)
1472	IGCM_debug_Exit "ERROR with OMP parameters !"
1473	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1474	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1475	IGCM_debug_Verif_Exit
1476	;;
1477	esac
1478	### only for LMDZ?
1479	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1480	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1481	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1482	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1483
1484	fi
1485
1486	# to have out/err per process on different files
1487	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1488	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1489
1490	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1491
1492	# Complete run_file
1493
1494	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1495
1496	fi
1497	done
1498
1499	## variable added to stop after 60s instead of 600s by default.
1500	## This is used when no error comes from executables and when something stopped an executable without notice.
1501	export SLURM_WAIT=60
1502
1503	## module advised by TGCC (instead of 2 variables)
1504	module load feature/bridge/heterogenous_mpmd
1505
1506	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1507
1508	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1509
1510	IGCM_sys_Chmod u+x run_file
1511	if ( $DEBUG_sys ) ; then
1512	echo "run_file contains : "
1513	cat run_file
1514	fi
1515
1516	else
1517
1518	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1519
1520	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1521
1522	else
1523	# Only one executable (SPMD mode).
1524
1525	for comp in ${config_ListOfComponents[*]} ; do
1526
1527	# Only if we really have an executable for the component :
1528	eval ExeNameOut=\${config_Executable_${comp}[1]}
1529	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1530
1531	# Build script files
1532
1533	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1534	echo "" >> script_${ExeNameOut}.ksh
1535	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1536
1537	if ( ${OK_PARA_OMP} ) ; then
1538	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1539	echo "" >> script_${ExeNameOut}.ksh
1540	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1541	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1542	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1543	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1544	fi
1545
1546	if ( ${OK_PARA_MPI} ) ; then
1547	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1548	# Default : ccc_mprun used if nb_proc gt 1
1549	# to have out/err per process on different files
1550	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1551	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1552	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1553	else
1554	# Default : ccc_mprun is NOT used if nb_proc eq 1
1555	# to have out/err per process on different files
1556	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1557	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1558	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1559	fi
1560
1561	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1562	cat script_${ExeNameOut}.ksh
1563
1564	fi
1565
1566	done
1567
1568	fi # ${OK_PARA_MPMD}
1569
1570	IGCM_debug_Print 1 "sys Irene : execution command is "
1571	IGCM_debug_Print 1 "$EXECUTION"
1572
1573	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1574	}
1575
1576	#D-#==================================================
1577	#D-function IGCM_sys_check_path
1578	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1579	#D-* do not point to an important use directory. Stop immediately in that case.
1580	#D-* Examples:
1581	#D-
1582	function IGCM_sys_check_path {
1583	IGCM_debug_PushStack "IGCM_sys_check_path"
1584	if ( $DEBUG_sys ) ; then
1585	echo "IGCM_sys_check_path"
1586	fi
1587
1588	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1589	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1590	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1591	IGCM_debug_Exit "This will stop the job"
1592	fi
1593	IGCM_debug_PopStack "IGCM_sys_check_path"
1594	}
1595
1596	#D-#==================================================
1597	#D-function IGCM_sys_check_quota
1598	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1599	#D-* Examples:
1600	#D-
1601	function IGCM_sys_check_quota {
1602	IGCM_debug_PushStack "IGCM_sys_check_quota"
1603	if ( $DEBUG_sys ) ; then
1604	echo "IGCM_sys_check_quota"
1605	fi
1606	# Limit of quota (in %)
1607	limit_quota=90
1608
1609	# Check of the volume
1610	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1611	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1612
1613	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1614
1615	unit_avail=${volume_avail: -1}
1616	unit_quota=${volume_quota: -1}
1617
1618	if [ "${unit_quota}" = "*" ] ; then
1619	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1620	IGCM_debug_Print 1 "More than 100% of your quota is used"
1621	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1622	IGCM_debug_Print 1 "You must have more than 10% available to run"
1623	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1624	IGCM_debug_Verif_Exit
1625	fi
1626
1627	temp_avail=${volume_avail%%${volume_avail: -1}*}
1628	temp_quota=${volume_quota%%${volume_quota: -1}*}
1629
1630	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1631
1632	# Convertion
1633	if [ ${volume_avail: -1} = "T" ] ; then
1634	(( temp_avail = temp_avail * 1000000000000 ))
1635	elif [ ${volume_avail: -1} = "G" ] ; then
1636	(( temp_avail = temp_avail * 1000000000 ))
1637	elif [ ${volume_avail: -1} = "M" ] ; then
1638	(( temp_avail = temp_avail * 1000000 ))
1639	elif [ ${volume_avail: -1} = "k" ] ; then
1640	(( temp_avail = temp_avail * 1000 ))
1641	else
1642	(( temp_avail = volume_avail ))
1643	fi
1644	if [ ${volume_quota: -1} = "T" ] ; then
1645	(( temp_quota = temp_quota * 1000000000000 ))
1646	elif [ ${volume_quota: -1} = "G" ] ; then
1647	(( temp_quota = temp_quota * 1000000000 ))
1648	elif [ ${volume_quota: -1} = "M" ] ; then
1649	(( temp_quota = temp_quota * 1000000 ))
1650	elif [ ${volume_quota: -1} = "k" ] ; then
1651	(( temp_quota = temp_quota * 1000 ))
1652	else
1653	(( temp_quota = volume_quota ))
1654	fi
1655	fi
1656
1657	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1658	# echo "volume ratio is " $quota_volume
1659
1660	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1661	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1662	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1663	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1664	IGCM_debug_Print 1 "You must have more than 10% available to run"
1665	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1666	IGCM_debug_Verif_Exit
1667	fi
1668
1669	fi
1670
1671	# Check of the number of inodes
1672
1673	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1674	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1675
1676	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1677
1678	unit_avail=${inode_avail: -1}
1679	unit_quota=${inode_quota: -1}
1680
1681	if [ "${unit_quota}" = "*" ] ; then
1682	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1683	IGCM_debug_Print 1 "More than 100% of your quota is used"
1684	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1685	IGCM_debug_Print 1 "You must have more than 10% available to run"
1686	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1687	IGCM_debug_Verif_Exit
1688	fi
1689
1690	temp_avail=${inode_avail%%${inode_avail: -1}*}
1691	temp_quota=${inode_quota%%${inode_quota: -1}*}
1692
1693	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1694
1695	# Convertion
1696	if [ ${inode_avail: -1} = "T" ] ; then
1697	(( temp_avail = temp_avail * 1000000000000 ))
1698	elif [ ${inode_avail: -1} = "G" ] ; then
1699	(( temp_avail = temp_avail * 1000000000 ))
1700	elif [ ${inode_avail: -1} = "M" ] ; then
1701	(( temp_avail = temp_avail * 1000000 ))
1702	elif [ ${inode_avail: -1} = "k" ] ; then
1703	(( temp_avail = temp_avail * 1000 ))
1704	else
1705	(( temp_avail = inode_avail ))
1706	fi
1707
1708	if [ ${inode_quota: -1} = "T" ] ; then
1709	(( temp_quota = temp_quota * 1000000000000 ))
1710	elif [ ${inode_quota: -1} = "G" ] ; then
1711	(( temp_quota = temp_quota * 1000000000 ))
1712	elif [ ${inode_quota: -1} = "M" ] ; then
1713	(( temp_quota = temp_quota * 1000000 ))
1714	elif [ ${inode_quota: -1} = "k" ] ; then
1715	(( temp_quota = temp_quota * 1000 ))
1716	else
1717	(( temp_quota = inode_quota ))
1718	fi
1719	fi
1720	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1721	# echo "inode ratio is " $quota_inode
1722
1723	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1724	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1725	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1726	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1727	IGCM_debug_Print 1 "You must have more than 10% available to run"
1728	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1729	IGCM_debug_Verif_Exit
1730	fi
1731	fi
1732	IGCM_debug_PopStack "IGCM_sys_check_quota"
1733	}
1734
1735	#D-#==================================================
1736	#D-function IGCM_sys_projectAccounting
1737	#D-* Purpose: store project accounting information in a file
1738	#D-* Examples:
1739	#D-
1740	function IGCM_sys_projectAccounting {
1741	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1742	if ( $DEBUG_sys ) ; then
1743	echo "IGCM_sys_check_quota"
1744	fi
1745
1746	ssh irene191 /usr/bin/ccc_myproject > $1
1747
1748	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1749	}
1750
1751	#D-#==================================================
1752	#D-function IGCM_sys_getJobSchedulerID
1753	#D-* Purpose: Get the job ID during execution
1754	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1755	#D-
1756	function IGCM_sys_getJobSchedulerID {
1757	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1758	if ( $DEBUG_sys ) ; then
1759	echo "IGCM_sys_getJobSchedulerID"
1760	fi
1761
1762	eval ${1}=${BRIDGE_MSUB_JOBID}
1763
1764	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1765	}
1766
1767	#D-#==================================================
1768	#D-function IGCM_sys_GetJobID
1769	#D-* Purpose: Get the job ID from the JobName
1770	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1771	#D-
1772	function IGCM_sys_GetJobID {
1773	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1774	if ( $DEBUG_sys ) ; then
1775	echo "IGCM_sys_GetJobID"
1776	fi
1777
1778	# With -f option, the full job name is given in the last column
1779	ID=$( ccc_mstat -f -u $2 \| \
1780	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1781
1782	eval ${3}=${ID}
1783	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1784	}
1785
1786	#D-#==================================================
1787	#D-function IGCM_sys_CountJobInQueue
1788	#D-* Purpose: Count number of users job
1789	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1790	#D-
1791	function IGCM_sys_CountJobInQueue {
1792	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1793	if ( $DEBUG_sys ) ; then
1794	echo "IGCM_sys_CountJobInQueue"
1795	fi
1796
1797	# With -f option, the full job name is given in the last column
1798	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1799
1800	eval ${2}=${NbRun}
1801
1802	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1803	}
1804
1805	#D-#==================================================
1806	#D-function IGCM_sys_ListJobInQueue
1807	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1808	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1809	#D-
1810	function IGCM_sys_ListJobInQueue {
1811	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1812	if ( $DEBUG_sys ) ; then
1813	echo "IGCM_sys_ListJobInQueue"
1814	fi
1815
1816	# With -f option, the full job name is given in the last column
1817	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1818	'( $2 == User && \
1819	$NF != /TS/ && \
1820	$NF !~ /PACK/ && \
1821	$NF !~ /REBUILD/ && \
1822	$NF !~ /pack/ ) \
1823	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1824
1825	eval set -A ${2} ${JobList[*]}
1826
1827	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1828	}
1829
1830	#D-#==================================================
1831	#D-function IGCM_sys_atlas
1832	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1833	#D-* Examples:
1834	#D-
1835	function IGCM_sys_atlas {
1836	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1837	if ( $DEBUG_sys ) ; then
1838	echo "IGCM_sys_atlas :" $@
1839	fi
1840
1841	typeset status
1842
1843	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1844	status=$?
1845	if [ ${status} -gt 0 ] ; then
1846	echo "IGCM_sys_atlas : error code ${status}"
1847	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1848	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1849	IGCM_debug_PopStack "IGCM_sys_atlas"
1850	return 1
1851	else
1852	IGCM_debug_PopStack "IGCM_sys_atlas"
1853	return 0
1854	fi
1855
1856	IGCM_debug_PopStack "IGCM_sys_atlas"
1857	}
1858
1859	#D-#==================================================
1860	#D-function IGCM_sys_rebuild_nemo
1861	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1862	#D-* Examples:
1863	#D-
1864
1865	function IGCM_sys_rebuild_nemo {
1866	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1867	if ( $DEBUG_sys ) ; then
1868	echo "IGCM_sys_rebuild_nemo :" $@
1869	fi
1870
1871	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1872
1873	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1874	}
1875

Note: See TracBrowser for help on using the repository browser.

Download in other formats: