Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_ada.ksh @ 1380

Last change on this file since 1380 was 1380, checked in by mafoipsl, 7 years ago
Add a function IGCM_sys_rebuild_nemo. On curie call $( ccc_home -u p86ipsl )/REBUILD_NEMO/rebuild_nemo. On other system call rebuild as usual. See also ticket #320.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 53.0 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sébastien Denvil
5	# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Ada
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host and user names
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=ada
71	# project name
72	typeset PROJECT=$(echo ${LOADL_STEP_GROUP:=NONE})
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=$( TZ=utc date -d '1970-01-01 '${wall_clock_limit} +%s )
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=llsubmit}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=ergon
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Set environment tools (ferret, nco, cdo, rebuild, ...)
93	#====================================================
94	source /smplocal/pub/Modules/default/init/ksh
95	if ( [ "X${LOADL_STEP_TYPE}" = "XPARALLEL" ] \|\| [ "X${LOADL_STEP_TYPE}" = "XSERIAL" ] ) ; then
96	module load python/2.7.10 > /dev/null 2>&1
97	. /smphome/rech/psl/rpsl035/.atlas_env_ada_intel_2013_0_bash > /dev/null 2>&1
98	else
99	module load python/2.7.10 > /dev/null 2>&1
100	module unload intel > /dev/null 2>&1
101	module load intel/2016.2 > /dev/null 2>&1
102	. /smphome/rech/psl/rpsl035/.atlas_env_ada_bash > /dev/null 2>&1
103	export PCMDI_MP=/workgpfs/rech/psl/rpsl035/PCMDI-MP
104	export UVCDAT_ANONYMOUS_LOG=no
105	fi
106	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
107	[ ! X${TaskType} = Xchecking ] && module list
108
109	export PATH=${PATH}:/smphome/rech/psl/rpsl035/AddNoise/src_X64_ADA/bin
110	export PATH=${PATH}:/smphome/rech/psl/rpsl035/AddPerturbation/src_X64_ADA/bin
111	export PATH=${PATH}:/smphome/rech/psl/rpsl035/bin/
112
113	#====================================================
114	# Host specific DIRECTORIES
115	#====================================================
116
117	#====================================================
118	#- MirrorlibIGCM for frontend
119	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
120
121	#====================================================
122	#- libIGCM_POST for frontend
123	typeset -r libIGCM_POST=${libIGCM}
124
125	#====================================================
126	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
127	typeset -r R_EXE="${MODIPSL}/bin"
128
129	#====================================================
130	#- SUBMIT_DIR : submission dir
131	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${LOADL_STEP_INITDIR}}
132
133	#====================================================
134	#- IN
135	typeset -r R_IN=${R_IN:=/workgpfs/rech/psl/rpsl035/IGCM}
136	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/u/rech/psl/rpsl376}
137
138	#====================================================
139	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
140	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${TMPDIR}}
141
142	#====================================================
143	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
144	typeset -r OUTCOMMAND_PATH=${RUN_DIR_PATH:-/tmp}
145
146	#====================================================
147	#- HOST_MPIRUN_COMMAND
148	if ( [ "X${LOADL_STEP_TYPE}" = "XPARALLEL" ] \|\| [ "X${LOADL_STEP_TYPE}" = "XSERIAL" ] ) ; then
149	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time poe"}
150	else
151	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time mpirun"}
152	fi
153	#====================================================
154	#- Max number of arguments passed to nco operator or demigration command
155	UNIX_MAX_LIMIT=360
156
157	#====================================================
158	#- set PackDefault to true on ada
159	PackDefault=true
160
161	#====================================================
162	#- Number of cores per node
163	core_per_node=32
164
165	#====================================================
166	#- Default number of MPI task for IPSL coupled model
167	#- required for backward compatibility
168	#-
169	DEFAULT_NUM_PROC_OCE=5
170	DEFAULT_NUM_PROC_CPL=1
171	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
172
173	#D-#==================================================
174	#D-function IGCM_sys_defineArchives
175	#D-* Purpose:
176	#D-* Define ARCHIVE : Dedicated to large files
177	#D-* Define STORAGE : Dedicated to small/medium files
178	#D-* Define R_OUT : Output tree located on ARCHIVE
179	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
180	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
181	#D-* if SpaceName=TEST everything is stored on WORKDIR
182	#D-* Examples:
183	#D-
184	function IGCM_sys_defineArchives {
185	IGCM_debug_PushStack "IGCM_sys_defineArchives"
186
187	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
188	#====================================================
189	#- ARCHIVE (dedicated to large files)
190	ARCHIVE=${config_UserChoices_ARCHIVE}
191	else
192	#====================================================
193	#- ARCHIVE (dedicated to large files)
194	ARCHIVE=$(echo ${HOME} \| sed -e "s,/.*home/,/arch/home/,")
195	fi
196
197	if [ ! X${config_UserChoices_STORAGE} = X ]; then
198	#====================================================
199	#- STORAGE (dedicated to small/medium files)
200	STORAGE=${config_UserChoices_STORAGE}
201	else
202	#====================================================
203	#- STORAGE (dedicated to small/medium files)
204	STORAGE=${WORKDIR}
205	fi
206
207	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
208	#====================================================
209	#- R_OUT
210	R_OUT=${WORKDIR}/IGCM_OUT
211
212	#====================================================
213	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
214	R_FIG=${WORKDIR}/IGCM_OUT
215
216	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
217
218	else
219	#====================================================
220	#- R_OUT
221	R_OUT=${ARCHIVE}/IGCM_OUT
222
223	#====================================================
224	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
225	R_FIG=${ARCHIVE}/IGCM_OUT
226	fi
227
228	#====================================================
229	#- R_BUF (ONLY FOR double copy an scratch)
230	R_BUF=${WORKDIR}/IGCM_OUT
231
232	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
233	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
234	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
235
236	IGCM_debug_PopStack "IGCM_sys_defineArchives"
237	}
238
239	#D-#==================================================
240	#D-function IGCM_sys_RshArchive
241	#D-* Purpose: Archive rsh command
242	#D-* Examples:
243	#D-
244	function IGCM_sys_RshArchive {
245	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
246	DEBUG_sys=false IGCM_sys_IsFileArchived "$@"
247	if [ $? = 0 ] ; then
248	rsh ${STOREHOST} exec /bin/ksh <<-EOF
249	${@}
250	EOF
251	status=$?
252	else
253	/bin/ksh <<-EOF
254	${@}
255	EOF
256	status=$?
257	fi
258	if [ ${status} -gt 0 ] ; then
259	IGCM_debug_Print 2 "IGCM_sys_RshArchive : rsh or command failed error code ${status}"
260	IGCM_debug_Exit "IGCM_sys_RshArchive"
261	fi
262	IGCM_debug_PopStack "IGCM_sys_RshArchive"
263	}
264
265	#D-#==================================================
266	#D-function IGCM_sys_RshArchive_NoError
267	#D-* Purpose: Archive rsh command, without error
268	#D-* used only in monitoring.job
269	#D-* Examples:
270	#D-
271	function IGCM_sys_RshArchive_NoError {
272	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
273	DEBUG_sys=false IGCM_sys_IsFileArchived "$@"
274	if [ $? = 0 ] ; then
275	rsh ${STOREHOST} exec /bin/ksh <<-EOF
276	${@} 2> /dev/null
277	EOF
278	else
279	/bin/ksh <<-EOF
280	${@} 2> /dev/null
281	EOF
282	fi
283	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
284	}
285
286	#D-#==================================================
287	#D-function IGCM_sys_MkdirArchive
288	#D-* Purpose: Mkdir on Archive
289	#D-* Examples:
290	#D-
291	function IGCM_sys_MkdirArchive {
292	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
293	if ( $DEBUG_sys ) ; then
294	echo "IGCM_sys_MkdirArchive :" $@
295	fi
296	#- creation de repertoire sur le serveur fichier
297	DEBUG_sys=false IGCM_sys_IsFileArchived $1
298	if [ $? = 0 ] ; then
299	rsh ${STOREHOST} -n mkdir -p $1
300	status=$?
301	else
302	mkdir -p $1
303	status=$?
304	fi
305
306	if [ ${status} -gt 0 ] ; then
307	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : rsh or mkdir failed error code ${status}"
308	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
309	fi
310	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
311	}
312
313	#D-#==================================================
314	#D-function IGCM_sys_TestDirArchive
315	#D-* Purpose: Test Directory that must exists on Archive
316	#D-* Examples:
317	#D-
318	function IGCM_sys_TestDirArchive {
319	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
320	if ( $DEBUG_sys ) ; then
321	echo "IGCM_sys_TestDirArchive :" $@
322	fi
323	typeset ExistFlag
324	#Command depends on targeted file system
325	DEBUG_sys=false IGCM_sys_IsFileArchived $1
326	if [ $? = 0 ] ; then
327	ExistFlag=$( IGCM_sys_RshArchive "[ -d $1 ] && echo 0 \|\| echo 1" )
328	else
329	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
330	fi
331	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
332	return ${ExistFlag}
333	}
334
335	#D-#==================================================
336	#D-function IGCM_sys_IsFileArchived
337	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
338	#D-* Examples:
339	#D-
340	function IGCM_sys_IsFileArchived {
341	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
342	if ( $DEBUG_sys ) ; then
343	echo "IGCM_sys_IsFileArchived :" $@
344	fi
345	typeset IsArchivedFlag
346	IsArchivedFlag=$( [ "X$( echo $@ \| grep \/arch\/home )" != "X" ] && echo 0 \|\| echo 1 )
347	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
348
349	return ${IsArchivedFlag}
350	}
351
352	#D-#==================================================
353	#D-function IGCM_sys_TestFileArchive
354	#D-* Purpose: Test file that must NOT EXISTS on Archive
355	#D-* Examples:
356	#D-
357	function IGCM_sys_TestFileArchive {
358	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
359	typeset ExistFlag
360	ExistFlag=$( IGCM_sys_RshArchive "[ -f $1 ] && echo 0 \|\| echo 1" )
361	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
362
363	return ${ExistFlag}
364	}
365
366	#D-#==================================================
367	#D-function IGCM_sys_CountFileArchive
368	#D-* Purpose: Count files on Archive filesystem
369	#D-* Examples:
370	#D-
371	function IGCM_sys_CountFileArchive {
372	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
373	#Command depends on targeted file system
374	DEBUG_sys=false IGCM_sys_IsFileArchived $1
375	if [ $? = 0 ] ; then
376	IGCM_sys_RshArchive "ls ${@} 2>/dev/null \| wc -l"
377	status=$?
378	else
379	ls ${@} 2>/dev/null \| wc -l
380	status=$?
381	fi
382	if [ ${status} -gt 0 ] ; then
383	echo "IGCM_sys_CountFileArchive : erreur."
384	fi
385	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
386	}
387
388	#D-#==================================================
389	#D-function IGCM_sys_Tree
390	#D-* Purpose: Tree directories with files on ${ARCHIVE}
391	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
392	#D-
393	function IGCM_sys_Tree {
394	IGCM_debug_PushStack "IGCM_sys_Tree" $@
395	if ( $DEBUG_sys ) ; then
396	echo "IGCM_sys_Tree :" $@
397	fi
398
399	\mfls -R $@
400
401	IGCM_debug_PopStack "IGCM_sys_Tree"
402	}
403
404	#D-#==================================================
405	#D-function IGCM_sys_Qsub
406	#D-* Purpose: Qsub new job
407	#D-* Examples:
408	#D-
409	function IGCM_sys_Qsub {
410	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
411	if ( $DEBUG_sys ) ; then
412	echo "IGCM_sys_Qsub :" $@
413	fi
414	typeset status
415
416	# We have to change output/error file
417	[ ${#@} = 1 ] && REP_FOR_JOB=${SUBMIT_DIR}
418	[ ${#@} = 2 ] && REP_FOR_JOB=${2}
419	sed -e "s:\# \@ output = .:\# \@ output = ${Script_Output}:" \
420	-e "s:\# \@ error = .:\# \@ error = ${Script_Output}:" \
421	$1 > ${REP_FOR_JOB}/JOB_FOR_IGCM
422	cd $REP_FOR_JOB ; /usr/bin/llsubmit JOB_FOR_IGCM > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1 ; status=$? ; cd - ;
423
424	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
425	if [ ${status} -gt 0 ] ; then
426	IGCM_debug_Print 2 "IGCM_sys_Qsub $1 : error code ${status}"
427	IGCM_debug_Exit "IGCM_sys_Qsub"
428	else
429	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ \| tail -1 \| sed -e s/\"//g )
430	IGCM_sys_Rm ${REP_FOR_JOB}/JOB_FOR_IGCM
431	fi
432	IGCM_debug_PopStack "IGCM_sys_Qsub"
433	}
434
435	#D-#==================================================
436	#D-function IGCM_sys_QsubPost
437	#D-* Purpose: Qsub new job on scalaire
438	#D-* Examples:
439	#D-
440	function IGCM_sys_QsubPost {
441	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
442	if ( $DEBUG_sys ) ; then
443	echo "IGCM_sys_QsubPost :" $@
444	fi
445	typeset status
446
447	# We have to change output/error file
448	[ ${#@} = 1 ] && REP_FOR_JOB=${POST_DIR}
449	[ ${#@} = 2 ] && REP_FOR_JOB=${2}
450
451	sed -e "s:\# \@ output = .:\# \@ output = ${Script_Post_Output}.out:" \
452	-e "s:\# \@ error = .:\# \@ error = ${Script_Post_Output}.out:" \
453	${libIGCM_POST}/$1.job > ${REP_FOR_JOB}/JOB_FOR_IGCM_$$
454
455	cd $REP_FOR_JOB ; /usr/bin/llsubmit JOB_FOR_IGCM_$$ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1 ; status=$? ; cd - ;
456
457	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
458	if [ ${status} -gt 0 ] ; then
459	IGCM_debug_Print 2 "IGCM_sys_QsubPost $1 : error code ${status}"
460	IGCM_debug_Exit "IGCM_sys_QsubPost"
461	else
462	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ \| tail -1 \| sed -e s/\"//g )
463	IGCM_sys_Rm ${REP_FOR_JOB}/JOB_FOR_IGCM_$$
464	fi
465	IGCM_debug_PopStack "IGCM_sys_QsubPost"
466	}
467
468	#D-*************************
469	#D- File transfer functions
470	#D-*************************
471	#D-
472
473	#D-#==================================================
474	#D-function IGCM_sys_RmRunDir
475	#D-* Purpose: rm tmpdir (dummy function most of the time batch
476	#D- scheduler will do the job)
477	#D-* Examples:
478	#D-
479	function IGCM_sys_RmRunDir {
480	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
481	if ( $DEBUG_sys ) ; then
482	echo "IGCM_sys_RmRunDir :" $@
483	IGCM_debug_Print 1 "Dummy call, let the scheduler do that."
484	fi
485	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
486	}
487
488	#D-#==================================================
489	#D-function IGCM_sys_Put_Dir
490	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
491	#D-* Examples:
492	#D-
493	function IGCM_sys_Put_Dir {
494	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
495	if ( $DEBUG_sys ) ; then
496	echo "IGCM_sys_Put_Dir :" $@
497	fi
498	if [ $DRYRUN = 0 ]; then
499	if [ ! -d ${1} ] ; then
500	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
501	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
502	return
503	fi
504
505	typeset status
506
507	# Only if we use rsync
508	#IGCM_sys_TestDirArchive $( dirname $2 )
509	#
510	#Command depends on targeted file system
511	DEBUG_sys=false IGCM_sys_IsFileArchived $2
512	if [ $? = 0 ] ; then
513	\rcp -r $1 ${STOREHOST}:$2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
514	status=$?
515	else
516	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
517	status=$?
518	fi
519
520	if [ ${status} -gt 0 ] ; then
521	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : rcp or cp failed error code ${status}"
522	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
523	IGCM_debug_Exit "IGCM_sys_Put_Dir"
524	else
525	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
526	fi
527	fi
528	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
529	}
530
531	#D-#==================================================
532	#D-function IGCM_sys_Get_Dir
533	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
534	#D-* Examples:
535	#D-
536	function IGCM_sys_Get_Dir {
537	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
538	if ( $DEBUG_sys ) ; then
539	echo "IGCM_sys_Get_Dir :" $@
540	fi
541	if [ $DRYRUN = 0 ]; then
542	typeset NB_ESSAI DELAI status i
543	# number of tentative
544	NB_ESSAI=3
545	# time delay between tentative
546	DELAI=2
547
548	# Only if we use rsync
549	#IGCM_sys_TestDirArchive $( dirname $2 )
550	#
551	# Command depends on targeted filesystem
552	DEBUG_sys=false IGCM_sys_IsFileArchived $1
553	if [ $? = 0 ] ; then
554	# add dmget (to demigrate all offline files) to reduce time of this command :
555	#IGCM_sys_RshArchive "dmfind $1 -state MIG -o -state OFL -o -state PAR \| dmget -q -n"
556	i=0
557	while [ $i -lt $NB_ESSAI ] ; do
558	\rcp -rp ${STOREHOST}:$1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
559	status=$?
560	if [ ${status} -gt 0 ]; then
561	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : rcp failed error code ${status} ${i}/${NB_ESSAI}"
562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
563	sleep $DELAI
564	else
565	break
566	fi
567	(( i = i + 1 ))
568	done
569	else
570	\cp -rp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
571	status=$?
572	if [ ${status} -gt 0 ] ; then
573	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
574	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
575	IGCM_debug_Exit "IGCM_sys_Get_Dir"
576	else
577	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
578	fi
579	fi
580	if [ ${status} -gt 0 ] ; then
581	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : (r)cp failed error code ${status}"
582	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
583	IGCM_debug_Exit "IGCM_sys_Get_Dir"
584	else
585	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
586	fi
587	fi
588	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
589	}
590
591	#D-#==================================================
592	#D-function IGCM_sys_Put_Rest
593	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
594	#D- File and target directory must exist.
595	#D-* Examples:
596	#D-
597	function IGCM_sys_Put_Rest {
598	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
599	if ( $DEBUG_sys ) ; then
600	echo "IGCM_sys_Put_Rest :" $@
601	fi
602	if [ $DRYRUN = 0 ]; then
603	if [ ! -f ${1} ] ; then
604	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
605	IGCM_debug_Exit "IGCM_sys_Put_Rest"
606	fi
607
608	typeset status
609	#
610	if [ X${JobType} = XRUN ] ; then
611	IGCM_sys_Chmod 444 ${1}
612	fi
613	#
614	#
615	#Command depends on targeted file system
616	DEBUG_sys=false IGCM_sys_IsFileArchived $2
617	if [ $? = 0 ] ; then
618	mfput $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
619	status=$?
620	else
621	IGCM_sys_MkdirArchive $( dirname $2 )
622	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
623	status=$?
624	fi
625
626	# #RSYNC WITH NETWORK RSH CALL
627	# IGCM_sys_MkdirArchive $( dirname $2 )
628	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
629	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
630
631	# #RSYNC WITH NFS USE
632	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
633	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
634
635	# status=$?
636	# IGCM_sys_Rsync_out $status
637
638	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
639	# (( status=status+$? ))
640
641	if [ ${status} -gt 0 ] ; then
642	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : mfput or cp failed error code ${status}"
643	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
644	IGCM_debug_Exit "IGCM_sys_Put_Rest"
645	else
646	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
647	fi
648	fi
649	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
650	}
651
652	#D-#==================================================
653	#D-function IGCM_sys_Put_Out
654	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
655	#D-* Examples:
656	#D-
657	function IGCM_sys_Put_Out {
658	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
659	if ( $DEBUG_sys ) ; then
660	echo "IGCM_sys_Put_Out :" $@
661	fi
662
663	typeset status
664
665	if [ $DRYRUN = 0 ]; then
666	if [ ! -f ${1} ] ; then
667	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
668	IGCM_debug_PopStack "IGCM_sys_Put_Out"
669	return 1
670	fi
671	#
672	if [ X${JobType} = XRUN ] ; then
673	if [ X${3} = X ] ; then
674	IGCM_sys_Chmod 444 ${1}
675	fi
676	fi
677	#
678	#
679	#Command depends on targeted file system
680	DEBUG_sys=false IGCM_sys_IsFileArchived $2
681	if [ $? = 0 ] ; then
682	mfput $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
683	status=$?
684	else
685	IGCM_sys_MkdirArchive $( dirname $2 )
686	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
687	status=$?
688	fi
689
690	# #RSYNC WITH NETWORK RSH CALL
691	# IGCM_sys_MkdirArchive $( dirname $2 )
692	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
693	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
694
695	# #RSYNC WITH NFS USE
696	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
697	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
698
699	# status=$?
700	# IGCM_sys_Rsync_out $status
701
702	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
703	# (( status=status+$? ))
704
705	if [ ${status} -gt 0 ] ; then
706	IGCM_debug_Print 2 "IGCM_sys_Put_Out : mfput or cp failed error code ${status}"
707	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
708	IGCM_debug_Exit "IGCM_sys_Put_Out"
709	else
710	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
711	fi
712	fi
713	IGCM_debug_PopStack "IGCM_sys_Put_Out"
714	return 0
715	}
716
717	#D-#==================================================
718	#D-function IGCM_sys_Get
719	#D-* Purpose: Get a file from ${ARCHIVE}
720	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
721	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
722	function IGCM_sys_Get {
723	IGCM_debug_PushStack "IGCM_sys_Get" $@
724
725	typeset DEST status dm_liste
726
727	if ( $DEBUG_sys ) ; then
728	echo "IGCM_sys_Get :" $@
729	fi
730	if [ $DRYRUN -le 2 ]; then
731	if [ X${1} = X'/l' ] ; then
732	eval set +A dm_liste \${${2}}
733	else
734	dm_liste=${1}
735	fi
736	eval DEST=\${${#}}
737
738	# test if the (first) file is present in the old computation :
739	DEBUG_sys=false IGCM_sys_IsFileArchived ${dm_liste[0]}
740	if [ $? = 0 ] ; then
741	IGCM_sys_TestFileArchive ${dm_liste[0]}
742	status=$?
743	else
744	IGCM_sys_TestFileBuffer ${dm_liste[0]}
745	status=$?
746	fi
747
748	if [ ${status} -gt 0 ] ; then
749	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
750	IGCM_debug_Exit "IGCM_sys_Get"
751	return
752	fi
753
754	#Command depends on targeted file system
755	DEBUG_sys=false IGCM_sys_IsFileArchived ${dm_liste[0]}
756	if [ $? = 0 ] ; then
757	mfget ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
758	status=$?
759	else
760	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
761	status=$?
762	fi
763
764	# #RSYNC WITH NETWORK RSH CALL
765	# echo ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
766	# ${RSYNC} ${RSYNC_opt} --rsync-path=${REMOTE_RSYNC} -e rsh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
767
768	# #RSYNC WITH NFS USE
769	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
770	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
771
772	# status=$?
773	# IGCM_sys_Rsync_out $status
774
775	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
776	# (( status=status+$? ))
777
778	if [ ${status} -gt 0 ] ; then
779	IGCM_debug_Print 2 "IGCM_sys_Get : mfget or cp failed error code ${status}"
780	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
781	IGCM_debug_Exit "IGCM_sys_Get"
782	else
783	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
784	fi
785	fi
786	IGCM_debug_PopStack "IGCM_sys_Get"
787	}
788
789	#D-#==================================================
790	#D-function IGCM_sys_GetDate_Monitoring
791	#D-* Purpose: get the last year for which the monitoring has been computed
792	#D-* Examples:
793	#D-
794	function IGCM_sys_GetDate_Monitoring {
795	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
796	if ( $DEBUG_sys ) ; then
797	echo "IGCM_sys_GetDate_Monitoring :" $@
798	fi
799
800	IGCM_sys_Cd /tmp
801	# NASTY HACK. We force /tmp here. This function is called only from the front-end and interactively.
802	RUN_DIR_PATH=/tmp IGCM_sys_Get ${1} .
803	eval ${2}=$( cdo showyear $( basename ${1} ) 2> /dev/null \| gawk '{ print $NF }' )
804	# Need to erase this file to avoid collision (permission denied getting file) between users on the front-end
805	IGCM_sys_Rm -f $( basename ${1} )
806	IGCM_sys_Cd -
807
808	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
809	}
810
811	#D-#==================================================
812	#D-function IGCM_sys_Put_Dods
813	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
814	#D-* Examples:
815	#D-
816	function IGCM_sys_Put_Dods {
817	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
818	if ( $DEBUG_sys ) ; then
819	echo "IGCM_sys_Put_Dods :" $@
820	fi
821	typeset status
822	if [ $DRYRUN = 0 ]; then
823	# We take our time on that
824	sleep 10
825	IGCM_sys_TestDirArchive ${R_SAVE}/${1}
826	if [ $? != 0 ] ; then
827	echo "WARNING : IGCM_sys_Put_Dods ${R_SAVE}/${1} DOES NOT EXIST ."
828	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
829	return
830	fi
831
832	rsh ${STOREHOST} exec /bin/ksh <<EOF
833	cd ${R_SAVE}
834	/arch/local/bin/dods_rm DODS/pub/${LOGIN}/${R_DODS}/${1} > /dev/null 2>&1
835	/bin/chmod -R u+w ${R_SAVE}/${1}
836	/arch/local/bin/dods_cp ${1} DODS/pub/${LOGIN}/${R_DODS} > /dev/null 2>&1
837	/bin/chmod -R +rX ${R_SAVE}/${1}
838	/bin/chmod -R u+w ${R_SAVE}/${1}
839	EOF
840	status=$?
841
842	if [ ${status} -gt 0 ] ; then
843	echo "IGCM_sys_Put_Dods : error."
844	IGCM_debug_Exit "IGCM_sys_Put_Dods"
845	fi
846	fi
847	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
848	}
849
850	##############################################################
851	# REBUILD OPERATOR
852
853	#D-#==================================================
854	#D-function IGCM_sys_sync
855	#D-* Purpose: flush buffer on disk (dummy function on Ada)
856	#D-* Examples:
857	#D-
858	function IGCM_sys_sync {
859	IGCM_debug_PushStack "IGCM_sys_sync" $@
860	if ( $DEBUG_sys ) ; then
861	echo "IGCM_sys_sync :" $@
862	echo "Dummy call, let the system do that."
863	fi
864	IGCM_debug_PopStack "IGCM_sys_sync"
865	}
866
867	############################################################
868	# Activate Running Environnment Variables
869
870	#D-#==================================================
871	#D-function IGCM_sys_activ_variables
872	#D-* Purpose: set environement variables prior to execution
873	#D-* Examples:
874	#D-
875	function IGCM_sys_activ_variables {
876	IGCM_debug_PushStack "IGCM_sys_activ_variables"
877	if ( $DEBUG_sys ) ; then
878	echo "IGCM_sys_activ_variables"
879	fi
880
881	# --------------------------------------------------------------------
882	#D- MPI specifications
883	# --------------------------------------------------------------------
884
885	# --------------------------------------------------------------------
886	#D- Other specifications
887	# --------------------------------------------------------------------
888
889	ulimit -s unlimited
890	## to be done only one time
891	## export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/smplocal/pub/NetCDF/4.1.3/lib:/smplocal/pub/HDF5/1.8.9/seq/lib
892	## echo ${LD_LIBRARY_PATH} \| grep -i netcdf >/dev/null 2>&1 \|\| export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/smplocal/pub/NetCDF/4.1.3/lib:/smplocal/pub/HDF5/1.8.9/seq/lib
893	IGCM_debug_Print 1 "set LD_LIBRARY_PATH=${LD_LIBRARY_PATH}"
894
895	export MP_STDOUTMODE=combined
896	IGCM_debug_Print 1 "set MP_STDOUTMODE=${MP_STDOUTMODE}"
897
898	## fix to reduce memory usage. Required since 2014/22/04. On ada IDRIS.
899	#export MP_EUILIBPATH=/smplocal/lib/ibmhpc/pe12012/ppe.pami/gnu/lib64/pami64
900	#IGCM_debug_Print 1 "set MP_EUILIBPATH=${MP_EUILIBPATH}"
901
902	IGCM_debug_PopStack "IGCM_sys_activ_variables"
903	}
904
905	############################################################
906	# Desactivate Running Environnment Variables
907
908	#D-#==================================================
909	#D-function IGCM_sys_desactiv_variables
910	#D-* Purpose: unset environement variables after execution
911	#D-* Examples:
912	#D-
913	function IGCM_sys_desactiv_variables {
914	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
915	if ( $DEBUG_sys ) ; then
916	echo "IGCM_sys_desactiv_variables"
917	fi
918	# --------------------------------------------------------------------
919	#D- MPI specifications
920	# --------------------------------------------------------------------
921
922	# --------------------------------------------------------------------
923	#D- Other specifications
924	# --------------------------------------------------------------------
925
926	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
927	}
928
929	############################################################
930	# Update job headers to be used by the scheduler
931
932	#D-#==================================================
933	#D-function IGCM_sys_updateHeaders
934	#D-* Purpose: Update job headers to be used by the scheduler
935	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
936	#D-
937	function IGCM_sys_updateHeaders {
938	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
939	if ( $DEBUG_sys ) ; then
940	echo "IGCM_sys_updateHeaders"
941	fi
942	typeset file
943	file=$1
944
945	if [ ${executionType} -eq 1 ] ; then
946	# MPMD + MPI
947	if [ X${MPIEnvironment} = XIBM ] ; then
948	sed -e "/::openMPthreads::/d" \
949	-e "s/::JobNumProcTot::/${coreNumber}/" \
950	${file} > ${file}.tmp
951	else
952	sed -e "/::openMPthreads::/d" \
953	-e "s/@ job_type = parallel/@ job_type = mpich/" \
954	-e "s/::JobNumProcTot::/${coreNumber}/" \
955	${file} > ${file}.tmp
956	fi
957
958	elif [ ${executionType} -eq 2 ] ; then
959	# MPMD + MPI + OMP
960	if [ X${MPIEnvironment} = XIBM ] ; then
961	sed -e "s/::openMPthreads::/${openMPthreads}/" \
962	-e "s/::JobNumProcTot::/${mpiTasks}/" \
963	${file} > ${file}.tmp
964	else
965	(( nodeNumber = coreNumber / core_per_node ))
966	[ $(( ${coreNumber} % ${core_per_node} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
967	sed -e "/::openMPthreads::/d" \
968	-e "s/@ job_type = parallel/@ job_type = mpich/" \
969	-e "s/@ total_tasks = ::JobNumProcTot::/@ node = ${nodeNumber} /" \
970	-e "/@ as_limit = 3.5gb/d" \
971	-e "s/::JobNumProcTot::/${mpiTasks}/" \
972	${file} > ${file}.tmp
973	fi
974
975	elif [ ${executionType} -eq 3 ] ; then
976	# SPMD + MPI/OMP
977	if [ X${MPIEnvironment} = XIBM ] ; then
978	sed -e "s/::openMPthreads::/${openMPthreads}/" \
979	-e "s/::JobNumProcTot::/${mpiTasks}/" \
980	${file} > ${file}.tmp
981	else
982	sed -e "s/::openMPthreads::/${openMPthreads}/" \
983	-e "s/@ job_type = parallel/@ job_type = mpich/" \
984	-e "s/::JobNumProcTot::/${mpiTasks}/" \
985	${file} > ${file}.tmp
986	fi
987
988	elif [ ${executionType} -eq 4 ] ; then
989	# SPMD + MPI only
990	if [ X${MPIEnvironment} = XIBM ] ; then
991	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
992	-e "/::openMPthreads::/d" \
993	${file} > ${file}.tmp
994	else
995	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
996	-e "s/@ job_type = parallel/@ job_type = mpich/" \
997	-e "/::openMPthreads::/d" \
998	${file} > ${file}.tmp
999	fi
1000
1001	elif [ ${executionType} -eq 5 ] ; then
1002	# SPMD + OMP only
1003	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1004	-e "s/@ job_type = parallel/@ job_type = serial/" \
1005	-e "/::JobNumProcTot::/d" \
1006	${file} > ${file}.tmp
1007
1008	elif [ ${executionType} -eq 6 ] ; then
1009	# SEQUENTIAL THEN
1010	sed -e "s/::JobNumProcTot::/1/" \
1011	-e "s/@ job_type = parallel/@ job_type = serial/" \
1012	-e "/::openMPthreads::/d" \
1013	${file} > ${file}.tmp
1014
1015	fi
1016
1017	IGCM_sys_Mv ${file}.tmp ${file}
1018
1019	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1020	}
1021
1022	############################################################
1023	# Build MPI/OMP scripts run file (dummy function)
1024
1025	#D-#==================================================
1026	#D-function IGCM_sys_build_run_file
1027	#D-* Purpose: build run file (deprecated)
1028	#D-* Examples:
1029	#D-
1030	function IGCM_sys_build_run_file {
1031
1032	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1033
1034	}
1035
1036	############################################################
1037	# Build MPI/OMP scripts
1038
1039	#D-#==================================================
1040	#D-function IGCM_sys_build_execution_scripts
1041	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1042	#D-* Examples:
1043	#D-
1044	function IGCM_sys_build_execution_scripts
1045	{
1046	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1047	if ( $DEBUG_sys ) ; then
1048	echo "IGCM_sys_build_execution_scripts " $@
1049	fi
1050
1051	EXECUTION=${HOST_MPIRUN_COMMAND}
1052
1053	# MPMD mode
1054	if ( ${OK_PARA_MPMD} ) ; then
1055
1056	# MPI IBM Environment
1057	if ( [ "X${LOADL_STEP_TYPE}" = "XPARALLEL" ] \|\| [ "X${LOADL_STEP_TYPE}" = "XSERIAL" ] ) ; then
1058	IGCM_debug_Print 1 "You use IBM MPI environment"
1059	if [ -f run_file ] ; then
1060	IGCM_sys_Rm -f run_file
1061	fi
1062	touch run_file
1063
1064	# Build run_file
1065
1066	# First loop on the components for the coupler ie oasis (only if oasis3)
1067	# the coupler ie oasis3 must be the first one
1068	for comp in ${config_ListOfComponents[*]} ; do
1069
1070	eval ExeNameIn=\${config_Executable_${comp}[0]}
1071	eval ExeNameOut=\${config_Executable_${comp}[1]}
1072
1073	# for CPL component only
1074	if [ "X${comp}" = "XCPL" ] ; then
1075
1076	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1077	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1078
1079	if ( ${OK_PARA_MPI} ) ; then
1080	(( mpi_count = 1 ))
1081	until [ ${mpi_count} -gt ${comp_proc_mpi_loc} ] ; do
1082	echo "./${ExeNameOut}" >> run_file
1083	(( mpi_count = mpi_count + 1 ))
1084	done
1085	else
1086	echo "./${ExeNameOut} " >> run_file
1087	fi
1088	fi
1089
1090	done
1091
1092	# Then second loop on the components
1093	for comp in ${config_ListOfComponents[*]} ; do
1094
1095	eval ExeNameIn=\${config_Executable_${comp}[0]}
1096	eval ExeNameOut=\${config_Executable_${comp}[1]}
1097
1098	# Only if we really have an executable for the component and not the coupler ie oasis:
1099	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1100
1101	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1102	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1103
1104	if ( ${OK_PARA_OMP} ) ; then
1105	# Check if the number of threads is correct
1106	case ${comp_proc_omp_loc} in
1107	1\|2\|4)
1108	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads on IBM MPI Environment"
1109	IGCM_debug_Print 1 "Beware : it may you use more CPU than needed."
1110	;;
1111	8)
1112	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads on IBM MPI Environment"
1113	IGCM_debug_Print 1 "Beware : it may you use more CPU than needed."
1114	IGCM_debug_Print 1 "Beware : these settings should not be used for IPSLCM coupled model!"
1115	;;
1116	16)
1117	IGCM_debug_Exit "ERROR with OMP parameters !"
1118	IGCM_debug_Print 2 "Beware : ${comp_proc_omp_loc} is too much for MPMD mode"
1119	IGCM_debug_Print 2 "You will use more CPU than needed : try to use Intel-MPI environment to do such a thing !"
1120	IGCM_debug_Verif_Exit
1121	;;
1122	*)
1123	IGCM_debug_Exit "ERROR with OMP parameters !"
1124	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1125	IGCM_debug_Print 2 "Only 1,2,4,8,16 as number of OMP threads are possible "
1126	IGCM_debug_Verif_Exit
1127	;;
1128	esac
1129
1130	fi
1131
1132	if ( ${OK_PARA_MPI} ) ; then
1133	(( mpi_count = 1 ))
1134	until [ ${mpi_count} -gt ${comp_proc_mpi_loc} ] ; do
1135	if ( ${OK_PARA_OMP} ) ; then
1136	echo "env OMP_NUM_THREADS=$comp_proc_omp_loc ./${ExeNameOut}" >> run_file
1137	else
1138	echo "./${ExeNameOut}" >> run_file
1139	fi
1140	(( mpi_count = mpi_count + 1 ))
1141	done
1142	else
1143	echo "./${ExeNameOut} " >> run_file
1144	fi
1145	fi
1146	done
1147	if ( ${OK_PARA_OMP} ) ; then
1148	export KMP_STACKSIZE=200m
1149	fi
1150
1151	EXECUTION="${HOST_MPIRUN_COMMAND} -pgmmodel mpmd -cmdfile ./run_file"
1152
1153	IGCM_sys_Chmod u+x run_file
1154	if ( $DEBUG_sys ) ; then
1155	echo "run_file contains : "
1156	cat run_file
1157	fi
1158
1159
1160	# MPI Intel Environment
1161	else
1162	IGCM_debug_Print 1 "You use Intel MPI environment"
1163
1164	# Only MPI (MPMD)
1165	if ( ! ${OK_PARA_OMP} ) ; then
1166	init_exec=n
1167
1168	# First loop on the components for the coupler ie oasis (only if oasis3)
1169	# the coupler ie oasis3 must be the first one
1170	for comp in ${config_ListOfComponents[*]} ; do
1171
1172	eval ExeNameIn=\${config_Executable_${comp}[0]}
1173	eval ExeNameOut=\${config_Executable_${comp}[1]}
1174
1175	# for CPL component only
1176	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1177
1178	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1179	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1180
1181	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1182	echo "" >> script_${ExeNameOut}.ksh
1183	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1184	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1185	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1186	init_exec=y
1187	fi
1188	done
1189	# Then second loop on the components
1190	for comp in ${config_ListOfComponents[*]} ; do
1191
1192	eval ExeNameIn=\${config_Executable_${comp}[0]}
1193	eval ExeNameOut=\${config_Executable_${comp}[1]}
1194
1195	# Only if we really have an executable for the component and not the coupler ie oasis:
1196	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1197
1198	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1199	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1200
1201	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1202	echo "" >> script_${ExeNameOut}.ksh
1203	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1204	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1205
1206	if [ ${init_exec} = y ] ; then
1207	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1208	else
1209	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1210	init_exec=y
1211	fi
1212	fi
1213	done
1214	# MPI-OpenMP (MPMD)
1215	else
1216
1217	# Execution specifications
1218	EXECUTION="${EXECUTION} -configfile run_file"
1219	export KMP_STACKSIZE=200m
1220	if [ -f run_file ] ; then
1221	IGCM_sys_Rm -f run_file
1222	fi
1223	touch run_file
1224
1225	# Initialisation of variables
1226	string_final=""
1227	string_final_hexa=""
1228	current_core=0
1229	executable_nb=1
1230
1231	# Hosts treatment
1232	for nodes in `echo $LOADL_PROCESSOR_LIST`
1233	do
1234	host[$i]=$nodes
1235	i=$((i+1))
1236	done
1237
1238	# Loop on the components
1239	for comp in ${config_ListOfComponents[*]} ; do
1240
1241	eval ExeNameIn=\${config_Executable_${comp}[0]}
1242	eval ExeNameOut=\${config_Executable_${comp}[1]}
1243
1244	# Not possible if oasis has an executable (i.e old version of oasis3)
1245	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1246	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1247	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1248	IGCM_debug_Verif_Exit
1249	fi
1250
1251	# Only if we really have an executable for the component :
1252	if [ "X${ExeNameOut}" != X\"\" ] ; then
1253
1254	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1255	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1256
1257	# Check if the number of threads is correct
1258	case ${comp_proc_omp_loc} in
1259	1\|2\|4\|8\|16)
1260	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1261	;;
1262	*)
1263	IGCM_debug_Exit "ERROR with OMP parameters !"
1264	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1265	IGCM_debug_Print 2 "Only 1,2,4,8,16 as number of OMP threads are possible "
1266	IGCM_debug_Verif_Exit
1267	;;
1268	esac
1269
1270
1271	# Build run_file for Ada Intel-MPI environment : method used to assign cores and nodes for the MPI process by using hexadecimal mask
1272	# Example of run_file :
1273	#-host host0 -n 4 -env I_MPI_PIN_DOMAIN=[ff,ff00,ff0000,ff000000] ./a.out
1274	#-host host1 -n 4 -env I_MPI_PIN_DOMAIN=[ff,ff00,ff0000,ff000000] ./a.out
1275	#-host host2 -n 10 -env I_MPI_PIN_DOMAIN=[3,c,30,c0,300,c00,3000,c000,30000,c0000,100000,200000,400000,800000,1000000,2000000,4000000,8000000,10000000,20000000] ./b.out
1276	#-host host2 -n 10 ./c.out
1277	# Example of final command :
1278	# mpirun -configfile run_file
1279
1280	rank=0
1281	# For one specific executable, loop on mpi process
1282	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1283	(( index_host = current_core / core_per_node ))
1284	host_value=${host[${index_host}]}
1285	(( slot = current_core % core_per_node ))
1286	# loop on omp threads for each mpi process (set the appropriate bit to 1 and append it to previous one)
1287	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1288	string_final="1"$string_final
1289	done
1290	# convert binary mask to hexadecimal mask
1291	if [ $rank -ne 0 ] ; then
1292	string_final_hexa=$string_final_hexa","$( printf '%x\n' "$((2#$string_final))" )
1293	else
1294	string_final_hexa=$( printf '%x\n' "$((2#$string_final))" )
1295	fi
1296	# replace bit 1 by bit 0 in order to append next one (next one wil be 1)
1297	string_final=$( echo $string_final \| sed "s/1/0/g" )
1298	# mpi rank = mpi_rank + 1
1299	(( rank = rank + 1 ))
1300	# current core takes into account the number of omp threads which was previously append
1301	(( current_core = current_core + comp_proc_omp_loc ))
1302	# We write to the configuration file either we switch to the next node or we switch to the next executable
1303	if ( [ $(( current_core / core_per_node )) -ne $index_host ] \|\| [ $nb_proc_mpi -eq $(($comp_proc_mpi_loc-1)) ] ) ; then
1304	# I_MPI_PIN_DOMAIN variable must be given once per node
1305	if [ $executable_nb -eq 1 ] ; then
1306	echo "-host $host_value -n $rank -env I_MPI_PIN_DOMAIN=[$string_final_hexa] ./$ExeNameOut" >> run_file
1307	else
1308	sed -i "/$host_value/s/\]/\,$string_final_hexa\]/g" run_file
1309	echo "-host $host_value -n $rank ./$ExeNameOut" >> run_file
1310	fi
1311	# +1 for the number of executbale on the same node
1312	if [ $nb_proc_mpi -eq $(($comp_proc_mpi_loc-1)) ] ; then
1313	(( executable_nb = executable_nb + 1 ))
1314	fi
1315	# Some initializations if we switch to the next node
1316	if [ $(( current_core / core_per_node )) -ne $index_host ] ; then
1317	string_final=""
1318	string_final_hexa=""
1319	rank=0
1320	executable_nb=1
1321	fi
1322	fi
1323	done
1324
1325	fi
1326	done
1327	IGCM_sys_Chmod u+x run_file
1328	if ( $DEBUG_sys ) ; then
1329	echo "run_file contains : "
1330	cat run_file
1331	fi
1332	fi
1333	fi
1334
1335	# Only one executable (SPMD mode).
1336	else
1337
1338	for comp in ${config_ListOfComponents[*]} ; do
1339
1340	# Only if we really have an executable for the component :
1341	eval ExeNameOut=\${config_Executable_${comp}[1]}
1342	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1343
1344	if ( ${OK_PARA_OMP} ) ; then
1345	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1346	export KMP_STACKSIZE=200m
1347	#export KMP_LIBRARY=turnaround
1348	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1349	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1350	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1351	fi
1352	if ( ${OK_PARA_MPI} ) ; then
1353	if ( [ "X${LOADL_STEP_TYPE}" = "XPARALLEL" ] \|\| [ "X${LOADL_STEP_TYPE}" = "XSERIAL" ] ) ; then
1354	EXECUTION="${HOST_MPIRUN_COMMAND} ./${ExeNameOut}"
1355	else
1356	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1357	EXECUTION="${HOST_MPIRUN_COMMAND} -np ${comp_proc_mpi_loc} ./${ExeNameOut}"
1358	fi
1359	else
1360	EXECUTION="/usr/bin/time ./${ExeNameOut}"
1361	fi
1362	fi
1363
1364	done
1365
1366	fi
1367
1368	IGCM_debug_Print 1 "sys ada : execution command is "
1369	IGCM_debug_Print 1 "$EXECUTION"
1370
1371	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1372	}
1373
1374	#D-#==================================================
1375	#D-function IGCM_sys_check_path
1376	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1377	#D-* do not point to an important use directory. Stop immediately in that case.
1378	#D-* Examples:
1379	#D-
1380	function IGCM_sys_check_path {
1381	IGCM_debug_PushStack "IGCM_sys_check_path"
1382	if ( $DEBUG_sys ) ; then
1383	echo "IGCM_sys_check_path"
1384	fi
1385
1386	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
1387	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1388	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1389	IGCM_debug_Exit "This will stop the job"
1390	fi
1391	IGCM_debug_PopStack "IGCM_sys_check_path"
1392	}
1393
1394	#D-#==================================================
1395	#D-function IGCM_sys_check_quota
1396	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1397	#D-* Examples:
1398	#D-
1399	function IGCM_sys_check_quota {
1400	IGCM_debug_PushStack "IGCM_sys_check_quota"
1401	if ( $DEBUG_sys ) ; then
1402	echo "IGCM_sys_check_quota"
1403	fi
1404	# Limit of quota (in %)
1405	limit_quota=90
1406
1407	# Check of the volume
1408	volume_quota=$(quota_u -w 2>/dev/null\| grep 'Quota soft' \| gawk '{print $5}')
1409	if [ ! X${volume_quota} = X ] ; then
1410	quota_volume=${volume_quota%%\%}
1411	# echo $quota_volume
1412	if [ $(echo "${quota_volume} > ${limit_quota}" \| bc) -eq 1 ] ; then
1413	IGCM_debug_Print 1 "Please, check your quota of volume on workgpfs"
1414	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1415	IGCM_debug_Print 1 "Use the quota_u -w command to check"
1416	IGCM_debug_Print 1 "You must have more than 10% available to run"
1417	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1418	IGCM_debug_Verif_Exit
1419	fi
1420
1421	fi
1422	IGCM_debug_PopStack "IGCM_sys_check_quota"
1423	}
1424
1425	#D-#==================================================
1426	#D-function IGCM_sys_projectAccounting
1427	#D-* Purpose: store project accounting information in a file
1428	#D-* Examples:
1429	#D-
1430	function IGCM_sys_projectAccounting {
1431	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1432	if ( $DEBUG_sys ) ; then
1433	echo "IGCM_sys_check_quota"
1434	fi
1435
1436	cpt > $1
1437
1438	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1439	}
1440
1441	#D-#==================================================
1442	#D-function IGCM_sys_getJobSchedulerID
1443	#D-* Purpose: Get the job ID during execution
1444	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1445	#D-
1446	function IGCM_sys_getJobSchedulerID {
1447	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1448	if ( $DEBUG_sys ) ; then
1449	echo "IGCM_sys_getJobSchedulerID"
1450	fi
1451
1452	eval ${1}=$( echo $LOADL_STEP_ID \| awk -F. '{print $4}' )
1453
1454	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1455	}
1456
1457	#D-#==================================================
1458	#D-function IGCM_sys_GetJobID
1459	#D-* Purpose: Get the job ID from the JobName
1460	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1461	#D-
1462	function IGCM_sys_GetJobID {
1463	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1464	if ( $DEBUG_sys ) ; then
1465	echo "IGCM_sys_GetJobID"
1466	fi
1467
1468	# Print only the full (-W) JobID (%id) and JobName (%jn)
1469	ID=$( llq -u $2 -W -f %id %jn \| \
1470	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1471
1472	eval ${3}=${ID}
1473	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1474	}
1475
1476	#D-#==================================================
1477	#D-function IGCM_sys_CountJobInQueue
1478	#D-* Purpose: Count number of users job
1479	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1480	#D-
1481	function IGCM_sys_CountJobInQueue {
1482	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1483	if ( $DEBUG_sys ) ; then
1484	echo "IGCM_sys_CountJobInQueue"
1485	fi
1486
1487	# Print only the full (-W) JobName (%jn)
1488	NbRun=$( llq -W -f %jn \| grep -c "$1" )
1489
1490	eval ${2}=${NbRun}
1491
1492	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1493	}
1494
1495	#D-#==================================================
1496	#D-function IGCM_sys_ListJobInQueue
1497	#D-* Purpose: Produce a list of users jobs
1498	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1499	#D-
1500	function IGCM_sys_ListJobInQueue {
1501	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1502	if ( $DEBUG_sys ) ; then
1503	echo "IGCM_sys_ListJobInQueue"
1504	fi
1505
1506	# With -W option, column width is as large as necessary
1507	set -A JobList $( llq -u $1 -W -f %jn \| head -n -2 \| tail -n +3 \| \
1508	gawk '( $1 != /TS/ && \
1509	$1 !~ /PACK/ && \
1510	$1 !~ /REBUILD/ && \
1511	$1 !~ /pack/ ) \
1512	{ print $1 }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1513
1514	eval set -A ${2} ${JobList[*]}
1515
1516	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1517	}
1518
1519	#D-#==================================================
1520	#D-function IGCM_sys_atlas
1521	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1522	#D-* Examples:
1523	#D-
1524	function IGCM_sys_atlas {
1525	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1526	if ( $DEBUG_sys ) ; then
1527	echo "IGCM_sys_atlas :" $@
1528	fi
1529
1530	typeset status
1531
1532	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1533	status=$?
1534	if [ ${status} -gt 0 ] ; then
1535	echo "IGCM_sys_atlas : error code ${status}"
1536	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1537	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1538	IGCM_debug_PopStack "IGCM_sys_atlas"
1539	return 1
1540	else
1541	IGCM_debug_PopStack "IGCM_sys_atlas"
1542	return 0
1543	fi
1544
1545	IGCM_debug_PopStack "IGCM_sys_atlas"
1546	}
1547
1548	#D-#==================================================
1549	#D-function IGCM_sys_rebuild_nemo
1550	#D-* Purpose: rebuild nemo parallel files with general rebuild
1551	#D-* Examples:
1552	#D-
1553
1554	function IGCM_sys_rebuild_nemo {
1555	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1556	if ( $DEBUG_sys ) ; then
1557	echo "IGCM_sys_rebuild_nemo :" $@
1558	fi
1559
1560	nemo_generic_restart_file_name_out=${1}
1561	nemo_extension_out=${3}
1562	shift ; shift ; shift
1563
1564	IGCM_sys_rebuild -o ${nemo_generic_restart_file_name_out}.${nemo_extension_out} $@
1565
1566	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1567	}
1568
1569

Note: See TracBrowser for help on using the repository browser.

Download in other formats: