Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_irene.ksh @ 1579

Last change on this file since 1579 was 1579, checked in by acosce, 14 months ago
Add management of redhat8 environment for irene skl
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 64.0 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Irene
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
41	typeset -r DRYRUN=0
42	typeset -r DRYRUN_DEBUG=4
43	else
44	typeset -r DRYRUN=${DRYRUN:=0}
45	typeset -r DRYRUN_DEBUG=0
46	fi
47
48
49	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
50	# -------------------------------------------------------------------------------------
51	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
52	# \| \| Cp/Exe/param/files \| \| \| \|
53	# \| \| Chmod Qsub \| \| \| \|
54	# -------------------------------------------------------------------------------------
55	# \| 0 \| yes \| yes \| yes \| yes \|
56	# -------------------------------------------------------------------------------------
57	# \| 1 \| yes \| yes \| yes \| no \|
58	# -------------------------------------------------------------------------------------
59	# \| 2 \| yes \| yes \| no \| no \|
60	# -------------------------------------------------------------------------------------
61	# \| 3 \| yes \| no \| no \| no \|
62	# -------------------------------------------------------------------------------------
63	# \| 4 \| yes \| yes \| creation of RUN_DIR and Job_debug \|
64	# -------------------------------------------------------------------------------------
65
66
67	#=====================================================
68	# Global Variables :
69	#=====================================================
70	# Language : "fr" or "en"
71	typeset -r MYLANG="fr"
72
73	#=====================================================
74	# Host user names project maxCpuTime
75	# $hostname ou hostname
76	typeset HOST=${HOST:=$( hostname )}
77	# $username ou whoami
78	typeset LOGIN=${LOGIN:=$( whoami )}
79	# $hostname of the MASTER job
80	typeset MASTER=irene
81	# add default project on irene
82	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=::default_project::} \| cut -d@ -f1 )
83	# jobWarningDelay in seconds
84	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
85
86	#D-
87	#D-#==================================================
88	#D-Program used in libIGCM
89	#D-#==================================================
90
91	# Submit command
92	typeset SUBMIT=${SUBMIT:=ccc_msub}
93	# rsync with path
94	typeset -r RSYNC=/usr/bin/rsync
95	# RSYNC_opt args to rsync
96	typeset -r RSYNC_opt="-va"
97	# ie storage filesystem
98	typeset -r STOREHOST=${MASTER}
99	typeset -r REMOTE_RSYNC=/usr/bin/rsync
100
101	#====================================================
102	# Set environment tools (ferret, nco, cdo, rebuild, ...)
103	#====================================================
104	typeset CCCOS=${CCCOS:=$( ccc_os )}
105
106	if [ X${TaskType} = Xcomputing ] ; then
107	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
108	else
109	module purge > /dev/null 2>&1
110	if [ X${CCCOS} = X"Rhel_8__x86_64" ] ; then
111	. $( ccc_home -u igcmg)/MachineEnvironment/irene_next/env_atlas_irene > /dev/null 2>&1
112	else
113	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
114	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
115	fi
116
117	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
118	export UVCDAT_ANONYMOUS_LOG=FALSE
119	fi
120
121	# FYI
122	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
123	[ ! X${TaskType} = Xchecking ] && module list
124
125	# For AddNoise and AddPertu3DOCE
126	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
127	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
128
129	# For rebuild
130	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
131	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
132
133	#====================================================
134	# Host specific DIRECTORIES
135	#====================================================
136
137	# ============ CESIUM START ============ #
138
139	#====================================================
140	#- Mirror libIGCM from titane to cesium if needed
141	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
142	#if [ ! ${ROOTSYS} = "home" ] ; then
143	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
144	#else
145	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
146	#fi
147
148	#====================================================
149	#- libIGCM_POST
150	#if ( ${MirrorlibIGCM} ) ; then
151	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
152	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
153	#else
154	# typeset -r libIGCM_POST=${libIGCM}
155	#fi
156
157	# ============ CESIUM END ============ #
158
159	#====================================================
160	#- MirrorlibIGCM for frontend
161	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
162
163	#====================================================
164	#- libIGCM_POST for frontend
165	typeset -r libIGCM_POST=${libIGCM}
166
167	#====================================================
168	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
169	typeset -r R_EXE="${MODIPSL}/bin"
170
171	#====================================================
172	#- SUBMIT_DIR : submission dir
173	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
174
175	#====================================================
176	#- IN
177	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
178
179	#====================================================
180	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
181	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
182
183	#====================================================
184	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
185	typeset -r OUTCOMMAND_PATH=/tmp
186
187	#====================================================
188	#- HOST_MPIRUN_COMMAND
189	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
190
191	#====================================================
192	#- Max number of arguments passed to nco operator or demigration command
193	UNIX_MAX_LIMIT=360
194
195	#====================================================
196	#- set PackDefault to true on irene
197	PackDefault=true
198
199	#====================================================
200	#- Default number of MPI task for IPSL coupled model
201	#- required for backward compatibility
202	#-
203	DEFAULT_NUM_PROC_OCE=5
204	DEFAULT_NUM_PROC_CPL=1
205	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
206
207	#====================================================
208	#- Number of computing cores per node
209	#-
210	NB_CORE_PER_NODE=48
211
212	#D-#==================================================
213	#D-function IGCM_sys_defineArchives
214	#D-* Purpose:
215	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
216	#D-* Define ARCHIVE : Dedicated to large files
217	#D-* Define STORAGE : Dedicated to small/medium files
218	#D-* Define R_OUT : Output tree located on ARCHIVE
219	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
220	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
221	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
222	#D-* Examples:
223	#D-
224	function IGCM_sys_defineArchives {
225	IGCM_debug_PushStack "IGCM_sys_defineArchives"
226
227	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
228	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
229	# Default option: Change the dfldatadir according to the project used for submission
230	# The variable DataProject in section UserChoices in config.card is not set or is set to DEFAULT
231	if [ X"$( echo ${PROJECT} \| grep cmip6 )" != "X" ] ; then
232	module switch dfldatadir dfldatadir/gencmip6
233	IGCM_debug_Print 1 "Load dfldatadir for forced gencmip6 project"
234	else
235	module switch dfldatadir dfldatadir/${PROJECT}
236	IGCM_debug_Print 1 "Load dfldatadir for project ${PROJECT}"
237	fi
238	else
239	# Use the project set by the variable DataProject in section UserChoices in config.card
240	module switch dfldatadir dfldatadir/${config_UserChoices_DataProject}
241	IGCM_debug_Print 1 "Load dfldatadir for DataProject red in config.card : ${config_UserChoices_DataProject}"
242	fi
243
244	#====================================================
245	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
246	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
247
248	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
249	#====================================================
250	#- ARCHIVE (dedicated to large files)
251	ARCHIVE=${config_UserChoices_ARCHIVE}
252	else
253	#====================================================
254	#- ARCHIVE (dedicated to large files)
255	ARCHIVE=${CCCSTOREDIR}
256	fi
257
258	if [ ! X${config_UserChoices_STORAGE} = X ]; then
259	#====================================================
260	#- STORAGE (dedicated to small/medium files)
261	STORAGE=${config_UserChoices_STORAGE}
262	else
263	#====================================================
264	#- STORAGE (dedicated to small/medium files)
265	STORAGE=${CCCWORKDIR}
266	fi
267
268	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
269	#====================================================
270	#- R_OUT
271	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
272
273	#====================================================
274	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
275	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
276
277	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
278
279	else
280	#====================================================
281	#- R_OUT
282	R_OUT=${ARCHIVE}/IGCM_OUT
283
284	#====================================================
285	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
286	R_FIG=${STORAGE}/IGCM_OUT
287	fi
288
289	#====================================================
290	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
291	CMIP6_BUF=${STORAGE}/IGCM_OUT
292
293	#====================================================
294	#- R_BUF (ONLY FOR double copy an scratch)
295	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
296
297	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
298	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
299	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
300
301	IGCM_debug_PopStack "IGCM_sys_defineArchives"
302	}
303
304	#D-#==================================================
305	#D-function IGCM_sys_RshArchive
306	#D-* Purpose: Archive rsh command
307	#D-* Examples:
308	#D-
309	function IGCM_sys_RshArchive {
310	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
311	/bin/ksh <<-EOF
312	${@}
313	EOF
314	status=$?
315	if [ ${status} -gt 0 ] ; then
316	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
317	IGCM_debug_Exit "IGCM_sys_RshArchive"
318	fi
319	IGCM_debug_PopStack "IGCM_sys_RshArchive"
320	}
321
322	#D-#==================================================
323	#D-function IGCM_sys_RshArchive_NoError
324	#D-* Purpose: Archive rsh command, without error
325	#D-* used only in monitoring.job
326	#D-* Examples:
327	#D-
328	function IGCM_sys_RshArchive_NoError {
329	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
330	/bin/ksh <<-EOF
331	${@} 2> /dev/null
332	EOF
333	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
334	}
335
336	#D-#==================================================
337	#D-function IGCM_sys_MkdirArchive
338	#D-* Purpose: Mkdir on Archive
339	#D-* Examples:
340	#D-
341	function IGCM_sys_MkdirArchive {
342	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
343	if ( $DEBUG_sys ) ; then
344	echo "IGCM_sys_MkdirArchive :" $@
345	fi
346	#- creation de repertoire sur le serveur fichier
347	if [ ! -d ${1} ]; then
348	\mkdir -p $1
349	status=$?
350
351	if [ ${status} -gt 0 ] ; then
352	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
353	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
354	fi
355	fi
356	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
357	}
358
359	#D-#==================================================
360	#D-function IGCM_sys_TestDirArchive
361	#D-* Purpose: Test Directory that must exists on Archive
362	#D-* Examples:
363	#D-
364	function IGCM_sys_TestDirArchive {
365	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
366	if ( $DEBUG_sys ) ; then
367	echo "IGCM_sys_TestDirArchive :" $@
368	fi
369	typeset ExistFlag
370	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
371	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
372	return ${ExistFlag}
373	}
374
375	#D-#==================================================
376	#D-function IGCM_sys_IsFileArchived
377	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
378	#D-* Examples:
379	#D-
380	function IGCM_sys_IsFileArchived {
381	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
382	if ( $DEBUG_sys ) ; then
383	echo "IGCM_sys_IsFileArchived :" $@
384	fi
385	typeset IsArchivedFlag
386	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
387	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
388
389	return ${IsArchivedFlag}
390	}
391
392	#D-#==================================================
393	#D-function IGCM_sys_TestFileArchive
394	#D-* Purpose: Test file that must NOT EXISTS on Archive
395	#D-* Examples:
396	#D-
397	function IGCM_sys_TestFileArchive {
398	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
399	typeset ExistFlag
400	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
401	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
402
403	return ${ExistFlag}
404	}
405
406	#D-#==================================================
407	#D-function IGCM_sys_CountFileArchive
408	#D-* Purpose: Count files on Archive filesystem
409	#D-* Examples:
410	#D-
411	function IGCM_sys_CountFileArchive {
412	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
413	ls ${@} 2>/dev/null \| wc -l
414	if [ $? -gt 0 ] ; then
415	echo "IGCM_sys_CountFileArchive : erreur."
416	fi
417	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
418	}
419
420	#D-#==================================================
421	#D-function IGCM_sys_Tree
422	#D-* Purpose: Tree directories with files on ${ARCHIVE}
423	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
424	#D-
425	function IGCM_sys_Tree {
426	IGCM_debug_PushStack "IGCM_sys_Tree" $@
427	if ( $DEBUG_sys ) ; then
428	echo "IGCM_sys_Tree :" $@
429	fi
430
431	\ls -lR ${@}
432
433	IGCM_debug_PopStack "IGCM_sys_Tree"
434	}
435
436	#D-#==================================================
437	#D-function IGCM_sys_Qsub
438	#D-* Purpose: Qsub new job
439	#D-* Examples:
440	#D-
441	function IGCM_sys_Qsub {
442	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
443	if ( $DEBUG_sys ) ; then
444	echo "IGCM_sys_Qsub :" $@
445	fi
446	typeset options status
447	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
448
449	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
450	status=$?
451
452	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
453	if [ ${status} -gt 0 ] ; then
454	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
455	IGCM_debug_Exit "IGCM_sys_Qsub"
456	else
457	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
458	fi
459	IGCM_debug_PopStack "IGCM_sys_Qsub"
460	}
461
462	#D-#==================================================
463	#D-function IGCM_sys_QsubPost
464	#D-* Purpose: Qsub new job on scalaire
465	#D-* Examples:
466	#D-
467	function IGCM_sys_QsubPost {
468	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
469	if ( $DEBUG_sys ) ; then
470	echo "IGCM_sys_QsubPost :" $@
471	fi
472	typeset options status
473	options="-Q normal -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
474
475	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
476	status=$?
477
478	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
479	if [ ${status} -gt 0 ] ; then
480	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
481	IGCM_debug_Exit "IGCM_sys_QsubPost"
482	else
483	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
484	fi
485	IGCM_debug_PopStack "IGCM_sys_QsubPost"
486	}
487
488	#D-*************************
489	#D- File transfer functions
490	#D-*************************
491	#D-
492
493	#D-#==================================================
494	#D-function IGCM_sys_RmRunDir
495	#D-* Purpose: rm tmpdir (dummy function most of the time batch
496	#D- scheduler will do the job)
497	#D-* Examples:
498	#D-
499	function IGCM_sys_RmRunDir {
500	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
501	if ( $DEBUG_sys ) ; then
502	echo "IGCM_sys_RmRunDir :" $@
503	fi
504
505	typeset status
506
507	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
508	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
509	status=$?
510
511	if [ ${status} -gt 0 ] ; then
512	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
513	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
514	IGCM_debug_Exit "IGCM_sys_RmRunDir"
515	else
516	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
517	fi
518	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
519	}
520
521	#D-#==================================================
522	#D-function IGCM_sys_Put_Dir
523	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
524	#D-* Examples:
525	#D-
526	function IGCM_sys_Put_Dir {
527	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
528	if ( $DEBUG_sys ) ; then
529	echo "IGCM_sys_Put_Dir :" $@
530	fi
531	if [ $DRYRUN = 0 ]; then
532	if [ ! -d ${1} ] ; then
533	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
534	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
535	return
536	fi
537
538	typeset status
539
540	# Only if we use rsync
541	#IGCM_sys_TestDirArchive $( dirname $2 )
542	#
543	#USUAL WAY
544	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
545	status=$?
546
547	if [ ${status} -gt 0 ] ; then
548	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
549	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
550	IGCM_debug_Exit "IGCM_sys_Put_Dir"
551	else
552	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
553	fi
554	fi
555	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
556	}
557
558	#D-#==================================================
559	#D-function IGCM_sys_Get_Dir
560	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
561	#D-* Examples:
562	#D-
563	function IGCM_sys_Get_Dir {
564	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
565	if ( $DEBUG_sys ) ; then
566	echo "IGCM_sys_Get_Dir :" $@
567	fi
568	if [ $DRYRUN = 0 ]; then
569	typeset NB_ESSAI DELAI status i
570	# number of tentative
571	NB_ESSAI=3
572	# time delay between tentative
573	DELAI=2
574
575	#
576	# USUAL WAY
577	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
578	ccc_hsm get -r $1
579
580	i=0
581	while [ $i -lt $NB_ESSAI ] ; do
582	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
583	status=$?
584	if [ ${status} -gt 0 ] ; then
585	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
586	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
587	sleep $DELAI
588	else
589	break
590	fi
591	(( i = i + 1 ))
592	done
593
594	if [ ${status} -gt 0 ] ; then
595	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
596	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
597	IGCM_debug_Exit "IGCM_sys_Get_Dir"
598	else
599	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
600	fi
601	fi
602	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
603	}
604
605	#D-#==================================================
606	#D-function IGCM_sys_Put_Rest
607	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
608	#D- File and target directory must exist.
609	#D-* Examples:
610	#D-
611	function IGCM_sys_Put_Rest {
612	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
613	if ( $DEBUG_sys ) ; then
614	echo "IGCM_sys_Put_Rest :" $@
615	fi
616	if [ $DRYRUN = 0 ]; then
617	if [ ! -f ${1} ] ; then
618	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
619	IGCM_debug_Exit "IGCM_sys_Put_Rest"
620	fi
621
622	typeset status
623	#
624	# USUAL WAY
625	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
626	status=$?
627
628	# #RSYNC WITH NETWORK SSH CALL
629	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
630	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
631
632	# #RSYNC WITH NFS USE
633	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
634	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
635
636	# status=$?
637	# IGCM_sys_Rsync_out $status
638
639	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
640	# (( status=status+$? ))
641
642	if [ ${status} -gt 0 ] ; then
643	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
644	[ -f ${1} ] && ls -l ${1}
645	[ -f ${2} ] && ls -l ${2}
646	[ -f ${2}/${1} ] && ls -l ${2}/${1}
647	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
648	IGCM_debug_Exit "IGCM_sys_Put_Rest"
649	else
650
651	if [ X${JobType} = XRUN ] ; then
652	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
653	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
654	fi
655
656	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
657	fi
658	fi
659	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
660	}
661
662	#D-#==================================================
663	#D-function IGCM_sys_Put_Out
664	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
665	#D-* Examples:
666	#D-
667	function IGCM_sys_Put_Out {
668	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
669	if ( $DEBUG_sys ) ; then
670	echo "IGCM_sys_Put_Out :" $@
671	fi
672
673	typeset NB_ESSAI DELAI status i exist skip
674	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
675
676	# number of tentative
677	NB_ESSAI=3
678	# time delay between tentative
679	DELAI=2
680
681	if [ $DRYRUN = 0 ]; then
682	if [ ! -f ${1} ] ; then
683	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
684	IGCM_debug_PopStack "IGCM_sys_Put_Out"
685	return 1
686	fi
687	#
688	IGCM_sys_MkdirArchive $( dirname $2 )
689	#
690	exist=false
691	skip=false
692	if [ -f $2 ] ; then
693	IGCM_debug_Print 1 "$2 already exist"
694	ccc_hsm get $2
695	exist=true
696	if [ "X$( diff $1 $2 )" = X ] ; then
697	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
698	skip=true
699	else
700	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
701	skip=false
702	fi
703	fi
704	#
705	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
706	IGCM_sys_Chmod u+w $2
707	fi
708
709	if [ X${skip} = Xfalse ] ; then
710	i=0
711	while [ $i -lt $NB_ESSAI ] ; do
712	# Identify file system
713	fileDeviceNumberInHex=$( stat -c %d $1 )
714	status=$?
715	if [ ${status} -gt 0 ] ; then
716	IGCM_debug_Exit "IGCM_sys_Put_Out"
717	fi
718	# Identify file system
719	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
720	status=$?
721	if [ ${status} -gt 0 ] ; then
722	IGCM_debug_Exit "IGCM_sys_Put_Out"
723	fi
724
725	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
726	# They are not on the same device. USUAL WAY
727	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
728	status=$?
729	else
730	# They are on the same device. NOT SO USUAL WAY
731	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
732	status=$?
733	fi
734	if [ ${status} -gt 0 ]; then
735	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
736	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
737	[ -f ${1} ] && ls -l ${1}
738	[ -f ${2} ] && ls -l ${2}
739	[ -f ${2}/${1} ] && ls -l ${2}/${1}
740	sleep $DELAI
741	else
742	break
743	fi
744	(( i = i + 1 ))
745	done
746	fi
747
748	# #RSYNC WITH NETWORK SSH CALL
749	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
750	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
751
752	# #RSYNC WITH NFS USE
753	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
754	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
755
756	# status=$?
757	# IGCM_sys_Rsync_out $status
758
759	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
760	# (( status=status+$? ))
761
762	if [ ${status} -gt 0 ] ; then
763	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
764	[ -f ${1} ] && ls -l ${1}
765	[ -f ${2} ] && ls -l ${2}
766	[ -f ${2}/${1} ] && ls -l ${2}/${1}
767	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
768	IGCM_debug_Exit "IGCM_sys_Put_Out"
769	else
770
771	if [ X${JobType} = XRUN ] ; then
772	if [ X${3} = X ] ; then
773	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
774	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
775	fi
776	fi
777
778	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
779	fi
780	fi
781	IGCM_debug_PopStack "IGCM_sys_Put_Out"
782	return 0
783	}
784
785	#D-#==================================================
786	#D-function IGCM_sys_Get
787	#D-* Purpose: Get a file from ${ARCHIVE}
788	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
789	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
790	function IGCM_sys_Get {
791	IGCM_debug_PushStack "IGCM_sys_Get" $@
792
793	typeset DEST dm_liste target file_work
794	typeset NB_ESSAI DELAI status i
795
796	if ( $DEBUG_sys ) ; then
797	echo "IGCM_sys_Get :" $@
798	fi
799
800	# number of tentative
801	NB_ESSAI=3
802	# time delay between tentative
803	DELAI=2
804
805	if [ $DRYRUN -le 2 ]; then
806	if [ X${1} = X'/l' ] ; then
807	eval set +A dm_liste \${${2}}
808	else
809	eval set +A dm_liste ${1}
810	fi
811	eval DEST=\${${#}}
812	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
813	status=$?
814	if [ ${status} -gt 0 ] ; then
815	echo "WARNING IGCM_sys_Get : error code ${status}"
816	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
817	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
818	fi
819
820	#if [ ${status} -gt 0 ] ; then
821	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
822	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
823	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
824	# sleep 30
825	# echo "We try another time"
826	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
827	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
828	# status=$?
829	# if [ ${status} -gt 0 ] ; then
830	# echo "ERROR IGCM_sys_Get : again demigration error :"
831	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
832	# IGCM_debug_Exit "IGCM_sys_Get"
833	# fi
834	# else
835	# echo "ERROR IGCM_sys_Get : demigration error :"
836	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
837	# IGCM_debug_Exit "IGCM_sys_Get"
838	# fi
839	#fi
840
841	# #RSYNC WITH NETWORK SSH CALL
842	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
843	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
844
845	# #RSYNC WITH NFS USE
846	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
847	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
848
849	# status=$?
850	# IGCM_sys_Rsync_out $status
851
852	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
853	# (( status=status+$? ))
854
855	#USUAL WAY
856	if [ X${1} = X'/l' ] ; then
857	for target in ${dm_liste[*]} ; do
858	local_file=$( basename ${target} )
859	# test if the target file is present before the loop
860	IGCM_sys_TestFileArchive ${target}
861	status=$?
862	if [ ${status} -gt 0 ] ; then
863	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
864	IGCM_debug_Exit "IGCM_sys_Get"
865	else
866	i=0
867	while [ $i -lt $NB_ESSAI ] ; do
868	#if [ X${DoLink} = Xtrue ] ; then
869	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
870	# status=$?
871	# else
872	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
873	# status=$?
874	#fi
875	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
876	status=$?
877	if [ ${status} -gt 0 ]; then
878	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
879	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
880	sleep $DELAI
881	else
882	break
883	fi
884	(( i = i + 1 ))
885	done
886	if [ ${status} -gt 0 ] ; then
887	echo "IGCM_sys_Get : error"
888	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
889	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
890	IGCM_debug_Exit "IGCM_sys_Get"
891	else
892	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
893	fi
894	fi
895	done
896	else
897	i=0
898	while [ $i -lt $NB_ESSAI ] ; do
899	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
900	status=$?
901	if [ ${status} -gt 0 ]; then
902	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
903	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
904	sleep $DELAI
905	else
906	break
907	fi
908	(( i = i + 1 ))
909	done
910	if [ ${status} -gt 0 ] ; then
911	echo "IGCM_sys_Get : error"
912	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
913	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
914	IGCM_debug_Exit "IGCM_sys_Get"
915	else
916	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
917	fi
918	fi
919	fi
920	IGCM_debug_PopStack "IGCM_sys_Get"
921	}
922
923	#D-#==================================================
924	#D-function IGCM_sys_GetDate_Monitoring
925	#D-* Purpose: get the last year for which the monitoring has been computed
926	#D-* Examples:
927	#D-
928	function IGCM_sys_GetDate_Monitoring {
929	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
930	if ( $DEBUG_sys ) ; then
931	echo "IGCM_sys_GetDate_Monitoring :" $@
932	fi
933
934	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
935
936	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
937	}
938
939	#D-#==================================================
940	#D-function IGCM_sys_Dods_Rm
941	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
942	#D-* Examples:
943	#D-
944	function IGCM_sys_Dods_Rm {
945	if ( $DEBUG_sys ) ; then
946	echo "IGCM_sys_Dods_Rm :" $@
947	fi
948	typeset status
949	if [ $DRYRUN = 0 ]; then
950
951	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
952	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
953	# echo "Nothing has been done."
954	# return
955	# fi
956
957	if [ "$#" -eq 1 ]; then
958	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
959	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
960	status=$?
961	else
962	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
963	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
964	status=$?
965	fi
966
967	# if [ ${status} -gt 0 ] ; then
968	# echo "IGCM_sys_Dods_Rm : error."
969	# cat out_dods_rm
970	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
971	# else
972	# rm out_dods_rm
973	# fi
974
975	fi
976	return $status
977	}
978
979	#D-#==================================================
980	#D-function IGCM_sys_Dods_Cp
981	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
982	#D-* Examples:
983	#D-
984	function IGCM_sys_Dods_Cp {
985	if ( $DEBUG_sys ) ; then
986	echo "IGCM_sys_Dods_Cp :" $@
987	fi
988	typeset status
989	if [ $DRYRUN = 0 ]; then
990
991	# if [ ! -d ${R_SAVE}/${1} ] ; then
992	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
993	# echo "Nothing has been done."
994	# return
995	# fi
996
997	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
998	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
999	status=$?
1000
1001	# if [ ${status} -gt 0 ] ; then
1002	# echo "IGCM_sys_Dods_Cp : error."
1003	# cat out_dods_cp
1004	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1005	# else
1006	# rm out_dods_cp
1007	# fi
1008
1009	fi
1010	return $status
1011	}
1012
1013	#D-#==================================================
1014	#D-function IGCM_sys_Put_Dods
1015	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1016	#D-* Examples:
1017	#D-
1018	function IGCM_sys_Put_Dods {
1019	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1020	if ( $DEBUG_sys ) ; then
1021	echo "IGCM_sys_Put_Dods :" $@
1022	fi
1023	typeset status
1024	if [ $DRYRUN = 0 ]; then
1025	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1026	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1027	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1028	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1029	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1030	return
1031	fi
1032
1033	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1034	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1035	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1036	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1037	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1038	return
1039	fi
1040	#
1041	if [ -d ${R_SAVE}/${1} ] ; then
1042	cd ${R_SAVE}
1043	elif [ -d ${R_FIGR}/${1} ] ; then
1044	cd ${R_FIGR}
1045	fi
1046
1047	IGCM_sys_Dods_Rm ${1}
1048	IGCM_sys_Dods_Cp ${1}
1049	status=0
1050
1051	if [ ${status} -gt 0 ] ; then
1052	echo "IGCM_sys_Put_Dods : error."
1053	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1054	fi
1055	fi
1056	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1057	}
1058
1059	##############################################################
1060	# REBUILD OPERATOR
1061
1062	#D-#==================================================
1063	#D-function IGCM_sys_sync
1064	#D-* Purpose: flush buffer on disk
1065	#D-* Examples:
1066	#D-
1067	function IGCM_sys_sync {
1068	IGCM_debug_PushStack "IGCM_sys_sync" $@
1069	if ( $DEBUG_sys ) ; then
1070	echo "IGCM_sys_sync :" $@
1071	fi
1072
1073	/bin/sync
1074
1075	IGCM_debug_PopStack "IGCM_sys_sync"
1076	}
1077
1078	############################################################
1079	# Activate Running Environnment Variables
1080
1081	#D-#==================================================
1082	#D-function IGCM_sys_activ_variables
1083	#D-* Purpose: set environement variables prior to execution
1084	#D-* Examples:
1085	#D-
1086	function IGCM_sys_activ_variables {
1087	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1088	if ( $DEBUG_sys ) ; then
1089	echo "IGCM_sys_activ_variables"
1090	fi
1091
1092	# --------------------------------------------------------------------
1093	#D- MODULE specifications
1094	# --------------------------------------------------------------------
1095	# Source the file EnvFile containing all module load needed to run the model.
1096	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1097	# it will be used. Otherwise a default file will be used.
1098	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1099	# EnvFile is set in config.card and the file exists
1100	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1101	EnvFile=${config_UserChoices_EnvFile}
1102	else
1103	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1104	module purge > /dev/null 2>&1
1105	if [ X${compilerVersion} = XV17 ] ; then
1106	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
1107	## EnvFile=/ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1108	EnvFile=$( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1109	else
1110	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
1111	fi
1112	fi
1113	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1114	IGCM_debug_Print 1 ${EnvFile}
1115	. ${EnvFile}
1116
1117	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Irene. "
1118	module list
1119
1120	# --------------------------------------------------------------------
1121	#D- MPI specifications
1122	# --------------------------------------------------------------------
1123
1124	# --------------------------------------------------------------------
1125	#D- Other specifications
1126	# --------------------------------------------------------------------
1127
1128	ulimit -s unlimited
1129
1130	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1131	}
1132
1133	############################################################
1134	# Desactivate Running Environnment Variables
1135
1136	#D-#==================================================
1137	#D-function IGCM_sys_desactiv_variables
1138	#D-* Purpose: unset environement variables after execution
1139	#D-* Examples:
1140	#D-
1141	function IGCM_sys_desactiv_variables {
1142	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1143	if ( $DEBUG_sys ) ; then
1144	echo "IGCM_sys_desactiv_variables"
1145	fi
1146	# --------------------------------------------------------------------
1147	#D- MPI specifications
1148	# --------------------------------------------------------------------
1149
1150	# --------------------------------------------------------------------
1151	#D- Other specifications
1152	# --------------------------------------------------------------------
1153
1154	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1155	}
1156
1157	############################################################
1158	# Update job headers to be used by the scheduler
1159
1160	#D-#==================================================
1161	#D-function IGCM_sys_updateHeaders
1162	#D-* Purpose: Update job headers to be used by the scheduler
1163	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1164	#D-
1165	function IGCM_sys_updateHeaders {
1166	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1167	if ( $DEBUG_sys ) ; then
1168	echo "IGCM_sys_updateHeaders"
1169	fi
1170	typeset file
1171	file=$1
1172
1173	if [ ${executionType} -eq 1 ] ; then
1174	# MPMD + MPI
1175	sed -e "/::openMPthreads::/d" \
1176	-e "s/::JobNumProcTot::/${coreNumber}/" \
1177	-e "/#MSUB -x/d" \
1178	-e "/--cpu_bind=none/d" \
1179	${file} > ${file}.tmp
1180
1181	elif [ ${executionType} -eq 2 ] ; then
1182	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1183	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1184	sed -e "/::openMPthreads::/d" \
1185	-e "s/::JobNumProcTot::/${coreNumber}/" \
1186	${file} > ${file}.tmp
1187	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1188	sed -e "/::openMPthreads::/d" \
1189	-e "s/::JobNumProcTot::/${coreNumber}/" \
1190	-e "/--cpu_bind=none/d" \
1191	${file} > ${file}.tmp
1192	else
1193	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1194	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1195	fi
1196	elif [ ${executionType} -eq 3 ] ; then
1197	# SPMD + MPI/OMP
1198	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1199	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1200	-e "/#MSUB -x/d" \
1201	-e "/--cpu_bind=none/d" \
1202	${file} > ${file}.tmp
1203
1204	elif [ ${executionType} -eq 4 ] ; then
1205	# SPMD + MPI only
1206	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1207	-e "/::openMPthreads::/d" \
1208	-e "/#MSUB -x/d" \
1209	-e "/--cpu_bind=none/d" \
1210	${file} > ${file}.tmp
1211
1212	elif [ ${executionType} -eq 5 ] ; then
1213	# SPMD + OMP only
1214	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1215	-e "/::JobNumProcTot::/d" \
1216	-e "/#MSUB -x/d" \
1217	-e "/--cpu_bind=none/d" \
1218	${file} > ${file}.tmp
1219
1220	elif [ ${executionType} -eq 6 ] ; then
1221	# SEQUENTIAL THEN
1222	sed -e "s/::JobNumProcTot::/1/" \
1223	-e "/::openMPthreads::/d" \
1224	-e "/#MSUB -x/d" \
1225	-e "/--cpu_bind=none/d" \
1226	${file} > ${file}.tmp
1227
1228	fi
1229
1230	IGCM_sys_Mv ${file}.tmp ${file}
1231
1232	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1233	}
1234
1235	############################################################
1236	# Build MPI/OMP scripts run file (dummy function)
1237
1238	#D-#==================================================
1239	#D-function IGCM_sys_build_run_file
1240	#D-* Purpose: build run file (deprecated)
1241	#D-* Examples:
1242	#D-
1243	function IGCM_sys_build_run_file {
1244	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1245	}
1246
1247	############################################################
1248	# Build MPI/OMP scripts
1249
1250	#D-#==================================================
1251	#D-function IGCM_sys_build_execution_scripts
1252	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1253	#D-* Examples:
1254	#D-
1255	function IGCM_sys_build_execution_scripts
1256	{
1257	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1258	if ( $DEBUG_sys ) ; then
1259	echo "IGCM_sys_build_execution_scripts " $@
1260	fi
1261
1262	EXECUTION=${HOST_MPIRUN_COMMAND}
1263
1264	if ( ${OK_PARA_MPMD} ) ; then
1265
1266	# MPMD mode
1267	# 1 MPI only
1268	# 2 MPI/OpenMP mpirun method
1269	# 3 MPI/OpenMP ccc_mprun method
1270
1271	if [ -f run_file ] ; then
1272	IGCM_sys_Rm -f run_file
1273	fi
1274	touch run_file
1275
1276	# case 1 : Only MPI (MPMD)
1277	if ( ! ${OK_PARA_OMP} ) ; then
1278
1279	# Build run_file
1280
1281	# First loop on the components for the coupler ie oasis (only if oasis3)
1282	# the coupler ie oasis3 must be the first one
1283	for comp in ${config_ListOfComponents[*]} ; do
1284
1285	eval ExeNameIn=\${config_Executable_${comp}[0]}
1286	eval ExeNameOut=\${config_Executable_${comp}[1]}
1287
1288	# for CPL component only
1289	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1290
1291	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1292	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1293	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1294	fi
1295	done
1296
1297	# Then second loop on the components
1298	for comp in ${config_ListOfComponents[*]} ; do
1299
1300	eval ExeNameIn=\${config_Executable_${comp}[0]}
1301	eval ExeNameOut=\${config_Executable_${comp}[1]}
1302
1303	# Only if we really have an executable for the component and not the coupler ie oasis:
1304	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1305
1306	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1307	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1308	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1309	fi
1310	done
1311
1312	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1313
1314	IGCM_sys_Chmod u+x run_file
1315	if ( $DEBUG_sys ) ; then
1316	echo "run_file contains : "
1317	cat run_file
1318	fi
1319
1320	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1321
1322	# 2 MPI/OpenMP mpirun method
1323
1324	# Use of mpirun instead of ccc_mprun
1325	EXECUTION="time mpirun"
1326
1327	# Hosts treatment
1328	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1329
1330	i=0
1331	rm -f hosts rankfile
1332	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1333	for nodes in `cat hosts.tmp`
1334	do
1335	host[$i]=$nodes
1336	echo "${host[$i]}" >> hosts
1337	IGCM_debug_Print 1 ${host[$i]}
1338	i=$((i+1))
1339	done
1340	rm -f hosts.tmp
1341
1342	listnodes=${host[*]}
1343
1344	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1345
1346	# Initialisation
1347	rank=0
1348	current_core=0
1349	init_exec=n
1350
1351	# Loop on the components
1352	for comp in ${config_ListOfComponents[*]} ; do
1353
1354	eval ExeNameIn=\${config_Executable_${comp}[0]}
1355	eval ExeNameOut=\${config_Executable_${comp}[1]}
1356
1357	# Not possible if oasis has an executable (i.e old version of oasis3)
1358	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1359	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1360	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1361	IGCM_debug_Verif_Exit
1362	fi
1363
1364	# Only if we really have an executable for the component :
1365	if [ "X${ExeNameOut}" != X\"\" ] ; then
1366
1367	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1368	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1369	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1370
1371	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1372	echo "" >> script_${ExeNameOut}.ksh
1373	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1374
1375	# Check if the number of threads is correct
1376	case ${comp_proc_omp_loc} in
1377	2\|4\|8\|16)
1378	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1379	;;
1380	*)
1381	IGCM_debug_Exit "ERROR with OMP parameters !"
1382	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1383	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1384	IGCM_debug_Verif_Exit
1385	;;
1386	esac
1387	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1388	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1389	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1390	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1391	fi
1392
1393	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1394	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1395	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1396	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1397	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1398
1399	if [ ${init_exec} = y ] ; then
1400	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1401	else
1402	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1403	init_exec=y
1404	fi
1405
1406	# Build rankfile : method used to assign cores and nodes for the MPI process
1407	# Ex :
1408	#rank 0=curie5296 slot=0,1,2,3
1409	#rank 1=curie5296 slot=4,5,6,7
1410	# Example of final command :
1411	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1412	# with script_lmdz.x.ksh :
1413	# #!/bin/ksh
1414	#export KMP_STACKSIZE=3g
1415	#export KMP_LIBRARY=turnaround
1416	#export MKL_SERIAL=YES
1417	#OMP_NUM_THREADS=4
1418	#./lmdz.x
1419	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1420	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1421	else
1422	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1423	fi
1424	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1425	(( index_host = current_core / NB_CORE_PER_NODE ))
1426	host_value=${host[${index_host}]}
1427	(( slot = current_core % NB_CORE_PER_NODE ))
1428	virg=","
1429	string_final=""
1430	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1431	string=$index$virg
1432	string_final=$string_final$string
1433	done
1434	string_final=$( echo $string_final \| sed "s/.$//" )
1435	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1436	(( rank = rank + 1 ))
1437	(( current_core = current_core + offset_comp_proc_loc ))
1438	done
1439	fi
1440	done
1441
1442	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1443
1444	# 3 MPI/OpenMP ccc_mprun method
1445
1446	# MPI-OpenMP (MPMD)
1447	# example of run_file
1448	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1449	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1450	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1451
1452	# Loop on the components
1453	for comp in ${config_ListOfComponents[*]} ; do
1454
1455	eval ExeNameIn=\${config_Executable_${comp}[0]}
1456	eval ExeNameOut=\${config_Executable_${comp}[1]}
1457
1458	# Not possible if oasis has an executable (i.e old version of oasis3)
1459	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1460	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1461	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1462	IGCM_debug_Verif_Exit
1463	fi
1464
1465	# Only if we really have an executable for the component :
1466	if [ "X${ExeNameOut}" != X\"\" ] ; then
1467
1468	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1469	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1470
1471	# Build script files
1472
1473	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1474	echo "" >> script_${ExeNameOut}.ksh
1475	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1476
1477	# Check if the number of threads is correct
1478	case ${comp_proc_omp_loc} in
1479	2\|4\|8\|12\|16\|24)
1480	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1481	;;
1482	*)
1483	IGCM_debug_Exit "ERROR with OMP parameters !"
1484	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1485	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1486	IGCM_debug_Verif_Exit
1487	;;
1488	esac
1489	### only for LMDZ?
1490	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1491	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1492	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1493	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1494
1495	fi
1496
1497	# to have out/err per process on different files
1498	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1499	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1500
1501	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1502
1503	# Complete run_file
1504
1505	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1506
1507	fi
1508	done
1509
1510	## variable added to stop after 60s instead of 600s by default.
1511	## This is used when no error comes from executables and when something stopped an executable without notice.
1512	export SLURM_WAIT=60
1513
1514	## module advised by TGCC (instead of 2 variables)
1515	module load feature/bridge/heterogenous_mpmd
1516
1517	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1518
1519	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1520
1521	IGCM_sys_Chmod u+x run_file
1522	if ( $DEBUG_sys ) ; then
1523	echo "run_file contains : "
1524	cat run_file
1525	fi
1526
1527	else
1528
1529	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1530
1531	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1532
1533	else
1534	# Only one executable (SPMD mode).
1535
1536	for comp in ${config_ListOfComponents[*]} ; do
1537
1538	# Only if we really have an executable for the component :
1539	eval ExeNameOut=\${config_Executable_${comp}[1]}
1540	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1541
1542	# Build script files
1543
1544	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1545	echo "" >> script_${ExeNameOut}.ksh
1546	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1547
1548	if ( ${OK_PARA_OMP} ) ; then
1549	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1550	echo "" >> script_${ExeNameOut}.ksh
1551	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1552	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1553	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1554	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1555	fi
1556
1557	if ( ${OK_PARA_MPI} ) ; then
1558	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1559	# Default : ccc_mprun used if nb_proc gt 1
1560	# to have out/err per process on different files
1561	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1562	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1563	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1564	else
1565	# Default : ccc_mprun is NOT used if nb_proc eq 1
1566	# to have out/err per process on different files
1567	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1568	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1569	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1570	fi
1571
1572	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1573	cat script_${ExeNameOut}.ksh
1574
1575	fi
1576
1577	done
1578
1579	fi # ${OK_PARA_MPMD}
1580
1581	IGCM_debug_Print 1 "sys Irene : execution command is "
1582	IGCM_debug_Print 1 "$EXECUTION"
1583
1584	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1585	}
1586
1587	#D-#==================================================
1588	#D-function IGCM_sys_check_path
1589	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1590	#D-* do not point to an important use directory. Stop immediately in that case.
1591	#D-* Examples:
1592	#D-
1593	function IGCM_sys_check_path {
1594	IGCM_debug_PushStack "IGCM_sys_check_path"
1595	if ( $DEBUG_sys ) ; then
1596	echo "IGCM_sys_check_path"
1597	fi
1598
1599	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1600	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1601	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1602	IGCM_debug_Exit "This will stop the job"
1603	fi
1604	IGCM_debug_PopStack "IGCM_sys_check_path"
1605	}
1606
1607	#D-#==================================================
1608	#D-function IGCM_sys_check_quota
1609	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1610	#D-* Examples:
1611	#D-
1612	function IGCM_sys_check_quota {
1613	IGCM_debug_PushStack "IGCM_sys_check_quota"
1614	if ( $DEBUG_sys ) ; then
1615	echo "IGCM_sys_check_quota"
1616	fi
1617	# Limit of quota (in %)
1618	limit_quota=90
1619
1620	# Check of the volume
1621	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1622	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1623
1624	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1625
1626	unit_avail=${volume_avail: -1}
1627	unit_quota=${volume_quota: -1}
1628
1629	if [ "${unit_quota}" = "*" ] ; then
1630	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1631	IGCM_debug_Print 1 "More than 100% of your quota is used"
1632	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1633	IGCM_debug_Print 1 "You must have more than 10% available to run"
1634	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1635	IGCM_debug_Verif_Exit
1636	fi
1637
1638	temp_avail=${volume_avail%%${volume_avail: -1}*}
1639	temp_quota=${volume_quota%%${volume_quota: -1}*}
1640
1641	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1642
1643	# Convertion
1644	if [ ${volume_avail: -1} = "T" ] ; then
1645	(( temp_avail = temp_avail * 1000000000000 ))
1646	elif [ ${volume_avail: -1} = "G" ] ; then
1647	(( temp_avail = temp_avail * 1000000000 ))
1648	elif [ ${volume_avail: -1} = "M" ] ; then
1649	(( temp_avail = temp_avail * 1000000 ))
1650	elif [ ${volume_avail: -1} = "k" ] ; then
1651	(( temp_avail = temp_avail * 1000 ))
1652	else
1653	(( temp_avail = volume_avail ))
1654	fi
1655	if [ ${volume_quota: -1} = "T" ] ; then
1656	(( temp_quota = temp_quota * 1000000000000 ))
1657	elif [ ${volume_quota: -1} = "G" ] ; then
1658	(( temp_quota = temp_quota * 1000000000 ))
1659	elif [ ${volume_quota: -1} = "M" ] ; then
1660	(( temp_quota = temp_quota * 1000000 ))
1661	elif [ ${volume_quota: -1} = "k" ] ; then
1662	(( temp_quota = temp_quota * 1000 ))
1663	else
1664	(( temp_quota = volume_quota ))
1665	fi
1666	fi
1667
1668	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1669	# echo "volume ratio is " $quota_volume
1670
1671	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1672	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1673	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1674	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1675	IGCM_debug_Print 1 "You must have more than 10% available to run"
1676	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1677	IGCM_debug_Verif_Exit
1678	fi
1679
1680	fi
1681
1682	# Check of the number of inodes
1683
1684	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1685	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1686
1687	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1688
1689	unit_avail=${inode_avail: -1}
1690	unit_quota=${inode_quota: -1}
1691
1692	if [ "${unit_quota}" = "*" ] ; then
1693	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1694	IGCM_debug_Print 1 "More than 100% of your quota is used"
1695	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1696	IGCM_debug_Print 1 "You must have more than 10% available to run"
1697	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1698	IGCM_debug_Verif_Exit
1699	fi
1700
1701	temp_avail=${inode_avail%%${inode_avail: -1}*}
1702	temp_quota=${inode_quota%%${inode_quota: -1}*}
1703
1704	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1705
1706	# Convertion
1707	if [ ${inode_avail: -1} = "T" ] ; then
1708	(( temp_avail = temp_avail * 1000000000000 ))
1709	elif [ ${inode_avail: -1} = "G" ] ; then
1710	(( temp_avail = temp_avail * 1000000000 ))
1711	elif [ ${inode_avail: -1} = "M" ] ; then
1712	(( temp_avail = temp_avail * 1000000 ))
1713	elif [ ${inode_avail: -1} = "k" ] ; then
1714	(( temp_avail = temp_avail * 1000 ))
1715	else
1716	(( temp_avail = inode_avail ))
1717	fi
1718
1719	if [ ${inode_quota: -1} = "T" ] ; then
1720	(( temp_quota = temp_quota * 1000000000000 ))
1721	elif [ ${inode_quota: -1} = "G" ] ; then
1722	(( temp_quota = temp_quota * 1000000000 ))
1723	elif [ ${inode_quota: -1} = "M" ] ; then
1724	(( temp_quota = temp_quota * 1000000 ))
1725	elif [ ${inode_quota: -1} = "k" ] ; then
1726	(( temp_quota = temp_quota * 1000 ))
1727	else
1728	(( temp_quota = inode_quota ))
1729	fi
1730	fi
1731	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1732	# echo "inode ratio is " $quota_inode
1733
1734	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1735	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1736	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1737	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1738	IGCM_debug_Print 1 "You must have more than 10% available to run"
1739	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1740	IGCM_debug_Verif_Exit
1741	fi
1742	fi
1743	IGCM_debug_PopStack "IGCM_sys_check_quota"
1744	}
1745
1746	#D-#==================================================
1747	#D-function IGCM_sys_projectAccounting
1748	#D-* Purpose: store project accounting information in a file
1749	#D-* Examples:
1750	#D-
1751	function IGCM_sys_projectAccounting {
1752	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1753	if ( $DEBUG_sys ) ; then
1754	echo "IGCM_sys_check_quota"
1755	fi
1756
1757	ssh irene191 /usr/bin/ccc_myproject > $1
1758
1759	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1760	}
1761
1762	#D-#==================================================
1763	#D-function IGCM_sys_getJobSchedulerID
1764	#D-* Purpose: Get the job ID during execution
1765	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1766	#D-
1767	function IGCM_sys_getJobSchedulerID {
1768	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1769	if ( $DEBUG_sys ) ; then
1770	echo "IGCM_sys_getJobSchedulerID"
1771	fi
1772
1773	eval ${1}=${BRIDGE_MSUB_JOBID}
1774
1775	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1776	}
1777
1778	#D-#==================================================
1779	#D-function IGCM_sys_GetJobID
1780	#D-* Purpose: Get the job ID from the JobName
1781	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1782	#D-
1783	function IGCM_sys_GetJobID {
1784	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1785	if ( $DEBUG_sys ) ; then
1786	echo "IGCM_sys_GetJobID"
1787	fi
1788
1789	# With -f option, the full job name is given in the last column
1790	ID=$( ccc_mstat -f -u $2 \| \
1791	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1792
1793	eval ${3}=${ID}
1794	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1795	}
1796
1797	#D-#==================================================
1798	#D-function IGCM_sys_CountJobInQueue
1799	#D-* Purpose: Count number of users job
1800	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1801	#D-
1802	function IGCM_sys_CountJobInQueue {
1803	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1804	if ( $DEBUG_sys ) ; then
1805	echo "IGCM_sys_CountJobInQueue"
1806	fi
1807
1808	# With -f option, the full job name is given in the last column
1809	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1810
1811	eval ${2}=${NbRun}
1812
1813	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1814	}
1815
1816	#D-#==================================================
1817	#D-function IGCM_sys_ListJobInQueue
1818	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1819	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1820	#D-
1821	function IGCM_sys_ListJobInQueue {
1822	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1823	if ( $DEBUG_sys ) ; then
1824	echo "IGCM_sys_ListJobInQueue"
1825	fi
1826
1827	# With -f option, the full job name is given in the last column
1828	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1829	'( $2 == User && \
1830	$NF != /TS/ && \
1831	$NF !~ /PACK/ && \
1832	$NF !~ /REBUILD/ && \
1833	$NF !~ /pack/ ) \
1834	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1835
1836	eval set -A ${2} ${JobList[*]}
1837
1838	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1839	}
1840
1841	#D-#==================================================
1842	#D-function IGCM_sys_atlas
1843	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1844	#D-* Examples:
1845	#D-
1846	function IGCM_sys_atlas {
1847	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1848	if ( $DEBUG_sys ) ; then
1849	echo "IGCM_sys_atlas :" $@
1850	fi
1851
1852	typeset status
1853
1854	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1855	status=$?
1856	if [ ${status} -gt 0 ] ; then
1857	echo "IGCM_sys_atlas : error code ${status}"
1858	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1859	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1860	IGCM_debug_PopStack "IGCM_sys_atlas"
1861	return 1
1862	else
1863	IGCM_debug_PopStack "IGCM_sys_atlas"
1864	return 0
1865	fi
1866
1867	IGCM_debug_PopStack "IGCM_sys_atlas"
1868	}
1869
1870	#D-#==================================================
1871	#D-function IGCM_sys_rebuild_nemo
1872	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1873	#D-* Examples:
1874	#D-
1875
1876	function IGCM_sys_rebuild_nemo {
1877	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1878	if ( $DEBUG_sys ) ; then
1879	echo "IGCM_sys_rebuild_nemo :" $@
1880	fi
1881
1882	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1883
1884	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1885	}
1886

Note: See TracBrowser for help on using the repository browser.

Download in other formats: