Context Navigation

← Previous Revision
Next Revision →
Normal
Revision Log

libIGCM_sys_obelix.ksh

Last change on this file was 1632, checked in by jgipsl, 4 months ago

Set jobWarningDelay so that the accounting mail can be done now also for obelix.

Correction in conversion from min to sec in sys_mesoipsl and sys_jeanzay since previous commit [1631].

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 41.7 KB

Rev	Line
[657]	1	#!/bin/ksh
[16]	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
[373]	5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
[16]	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
[657]	11	#
[16]	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[657]	16	# if you use the prefix #D- for comments to be extracted.
[16]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
	21	#D-LibIGCM_sys for obelix
	22	#D-#==================================================
	23	#D-
[657]	24	#D- This ksh library if a layer under some usefull
[16]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
[343]	34	#====================================================
[16]	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
[343]	38	#====================================================
[16]	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
[1569]	40	if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
	41	typeset -r DRYRUN=0
	42	typeset -r DRYRUN_DEBUG=4
	43	else
	44	typeset -r DRYRUN=${DRYRUN:=0}
	45	typeset -r DRYRUN_DEBUG=0
	46	fi
[16]	47
	48	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	49	# -------------------------------------------------------------------------------------
	50	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	51	# \| \| Cp/Exe/param/files \| \| \| \|
	52	# \| \| Chmod Qsub \| \| \| \|
[16]	53	# -------------------------------------------------------------------------------------
	54	# \| 0 \| yes \| yes \| yes \| yes \|
	55	# -------------------------------------------------------------------------------------
	56	# \| 1 \| yes \| yes \| yes \| no \|
	57	# -------------------------------------------------------------------------------------
	58	# \| 2 \| yes \| yes \| no \| no \|
	59	# -------------------------------------------------------------------------------------
	60	# \| 3 \| yes \| no \| no \| no \|
	61	# -------------------------------------------------------------------------------------
[1569]	62	# \| 4 \| yes \| yes \| creation of RUN_DIR and Job_debug \|
	63	# -------------------------------------------------------------------------------------
[16]	64
[1569]	65
[16]	66	#=====================================================
	67	# Global Variables :
	68	#=====================================================
	69	# Language : "fr" or "en"
	70	typeset -r MYLANG="fr"
	71
	72	#=====================================================
[1300]	73	# Host user names project maxCpuTime
[217]	74	# $hostname ou hostname
[1180]	75	typeset HOST=${HOST:=$( hostname )}
[16]	76	# $username ou whoami
[1180]	77	typeset LOGIN=${LOGIN:=$( whoami )}
[657]	78	# $hostname of the MASTER job
[742]	79	typeset MASTER=${MASTER:=$( hostname )}
[1201]	80	# project name
	81	typeset PROJECT=NONE
	82	# jobWarningDelay in seconds
[1632]	83	typeset jobWarningDelayHour=$(qstat $PBS_JOBID -f \| grep Resource_List.walltime \| awk -F: '{print $1}' \| awk -F= '{print $2}')
	84	typeset jobWarningDelay=$( echo "scale=6;${jobWarningDelayHour=}*3600" \| bc )
[16]	85	#D-
	86	#D-#==================================================
	87	#D-Program used in libIGCM
	88	#D-#==================================================
	89
[742]	90	# Submit command
	91	typeset SUBMIT=${SUBMIT:=qsub}
[16]	92	# rsync with path
	93	typeset -r RSYNC=/usr/bin/rsync
	94	# RSYNC_opt args to rsync
[282]	95	typeset -r RSYNC_opt="-va"
[657]	96	# ie storage filesystem
	97	typeset -r STOREHOST=${MASTER}
[16]	98
	99	#====================================================
[1485]	100	# Access to module command
	101	#====================================================
	102	. /usr/share/Modules/init/ksh
	103
	104	#====================================================
[1180]	105	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[305]	106	#====================================================
[1485]	107	if [ X${TaskType} = Xcomputing ] ; then
	108	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
	109	else
	110	. /home/orchideeshare/igcmg/MachineEnvironment/obelix/env_atlas_obelix
	111	fi
	112
	113	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
	114	[ ! X${TaskType} = Xchecking ] && module list
	115
	116	# Load python
[1358]	117	module load python/2.7.5
[16]	118
[343]	119	#====================================================
[16]	120	# Host specific DIRECTORIES
	121	#====================================================
[1485]	122	# For rebuild
[1594]	123	export PATH=${PATH}:${MODIPSL}/modeles/IOIPSL/bin:/home/orchideeshare/igcmg/IOIPSL_PLUS/modipsl.tagv2_2_4.28082019/bin/
[16]	124
	125	#====================================================
[657]	126	#- MirrorlibIGCM for frontend
	127	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	128
	129	#====================================================
	130	#- libIGCM_POST for frontend
	131	typeset -r libIGCM_POST=${libIGCM}
	132
	133	#====================================================
[16]	134	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	135	typeset -r R_EXE="${MODIPSL}/bin"
	136
	137	#====================================================
	138	#- SUBMIT_DIR : submission dir
[256]	139	if [ X${PBS_O_WORKDIR} != X ] ; then
[1124]	140	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
[256]	141	else
[1124]	142	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
[256]	143	fi
[657]	144
[16]	145	#====================================================
[725]	146	#- IN
[978]	147	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
[725]	148
	149	#====================================================
[16]	150	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1601]	151	if [ X${PBS_JOBID} != X ] ; then
	152	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/scratch01/$LOGIN/RUN_DIR/$PBS_O_LOGNAME.$PBS_JOBID}
[838]	153	else
[1601]	154	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/scratch01/$LOGIN/RUN_DIR/tmp$$}
[838]	155	fi
[16]	156
[1601]	157
[16]	158	#====================================================
[1179]	159	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	160	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	161
	162	#====================================================
[16]	163	#- HOST_MPIRUN_COMMAND
	164	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
	165
[61]	166	#====================================================
	167	#- Max number of arguments passed to nco operator or demigration command
	168	UNIX_MAX_LIMIT=360
	169
[657]	170	#====================================================
[1180]	171	#- set PackDefault to false on obelix
	172	PackDefault=false
	173
	174	#====================================================
[657]	175	#- Default number of MPI task for IPSL coupled model
	176	#- required for backward compatibility
	177	#-
[1300]	178	DEFAULT_NUM_PROC_OCE=5
[657]	179	DEFAULT_NUM_PROC_CPL=1
	180	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	181
[16]	182	#D-#==================================================
[1057]	183	#D-function IGCM_sys_defineArchives
	184	#D-* Purpose:
	185	#D-* Define ARCHIVE : Dedicated to large files
	186	#D-* Define STORAGE : Dedicated to small/medium files
	187	#D-* Define R_OUT : Output tree located on ARCHIVE
	188	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	189	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
	190	#D-* if SpaceName=TEST nothing special will hapen
[725]	191	#D-* Examples:
	192	#D-
[1057]	193	function IGCM_sys_defineArchives {
	194	IGCM_debug_PushStack "IGCM_sys_defineArchives"
	195
[1060]	196	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	197	#====================================================
	198	#- ARCHIVE (dedicated to large files)
[1063]	199	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	200	else
	201	#====================================================
	202	#- ARCHIVE (dedicated to large files)
[1063]	203	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
[1057]	204	fi
	205
[1060]	206	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	207	#====================================================
	208	#- STORAGE (dedicated to small/medium files)
[1063]	209	STORAGE=${config_UserChoices_STORAGE}
[1057]	210	else
	211	#====================================================
	212	#- STORAGE (dedicated to small/medium files)
[1063]	213	STORAGE=${ARCHIVE}
[1057]	214	fi
	215
	216	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
	217
	218	#====================================================
	219	#- R_OUT
[1063]	220	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	221
	222	#====================================================
	223	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	224	R_FIG=${STORAGE}/IGCM_OUT
[1057]	225
	226	#====================================================
[1431]	227	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	228	CMIP6_BUF=${STORAGE}/IGCM_OUT
	229
	230	#====================================================
[1057]	231	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	232	R_BUF=${STORAGE}/IGCM_OUT
[1057]	233
	234	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	235	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	236	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	237
	238	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	239	}
	240
	241	#D-#==================================================
[37]	242	#D-function IGCM_sys_RshArchive
	243	#D-* Purpose: Archive rsh command
	244	#D-* Examples:
	245	#D-
	246	function IGCM_sys_RshArchive {
[657]	247	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	248	/bin/ksh <<-EOF
[40]	249	${@}
	250	EOF
[811]	251	status=$?
	252	if [ ${status} -gt 0 ] ; then
	253	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[657]	254	IGCM_debug_Exit "IGCM_sys_RshArchive"
	255	fi
	256	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[37]	257	}
	258
	259	#D-#==================================================
[1178]	260	#D-function IGCM_sys_RshArchive_NoError
	261	#D-* Purpose: Archive rsh command, without error
	262	#D-* used only in monitoring.job
	263	#D-* Examples:
	264	#D-
	265	function IGCM_sys_RshArchive_NoError {
	266	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	267	/bin/ksh <<-EOF
[1181]	268	${@} 2> /dev/null
[1178]	269	EOF
	270	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	271	}
	272
	273	#D-#==================================================
[16]	274	#D-function IGCM_sys_MkdirArchive
	275	#D-* Purpose: Mkdir on Archive
	276	#D-* Examples:
	277	#D-
	278	function IGCM_sys_MkdirArchive {
[657]	279	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	280	if ( $DEBUG_sys ) ; then
	281	echo "IGCM_sys_MkdirArchive :" $@
	282	fi
[717]	283	#- creation de repertoire sur le serveur fichier
[657]	284	if [ ! -d ${1} ]; then
[1178]	285	\mkdir -p $1
[811]	286	status=$?
	287
	288	if [ ${status} -gt 0 ] ; then
	289	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[657]	290	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[16]	291	fi
[657]	292	fi
	293	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[16]	294	}
	295
	296	#D-#==================================================
	297	#D-function IGCM_sys_TestDirArchive
	298	#D-* Purpose: Test Directory that must exists on Archive
	299	#D-* Examples:
	300	#D-
	301	function IGCM_sys_TestDirArchive {
[657]	302	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	303	if ( $DEBUG_sys ) ; then
	304	echo "IGCM_sys_TestDirArchive :" $@
	305	fi
	306	typeset ExistFlag
	307	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	308	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	309	return ${ExistFlag}
[16]	310	}
	311
	312	#D-#==================================================
[800]	313	#D-function IGCM_sys_IsFileArchived
[1180]	314	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[800]	315	#D-* Examples:
	316	#D-
	317	function IGCM_sys_IsFileArchived {
	318	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	319	if ( $DEBUG_sys ) ; then
	320	echo "IGCM_sys_IsFileArchived :" $@
	321	fi
	322	typeset IsArchivedFlag
	323	# Never archived for this system
	324	IsArchivedFlag=1
	325	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	326
	327	return ${IsArchivedFlag}
	328	}
	329
	330	#D-#==================================================
[16]	331	#D-function IGCM_sys_TestFileArchive
	332	#D-* Purpose: Test file that must NOT EXISTS on Archive
	333	#D-* Examples:
	334	#D-
	335	function IGCM_sys_TestFileArchive {
[657]	336	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	337	typeset ExistFlag
	338	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	339	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[67]	340
[657]	341	return ${ExistFlag}
[67]	342	}
	343
	344	#D-#==================================================
	345	#D-function IGCM_sys_CountFileArchive
	346	#D-* Purpose: Count files on Archive filesystem
	347	#D-* Examples:
	348	#D-
	349	function IGCM_sys_CountFileArchive {
[657]	350	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	351	ls ${@} 2>/dev/null \| wc -l
	352	if [ $? -gt 0 ] ; then
	353	echo "IGCM_sys_CountFileArchive : erreur."
	354	fi
	355	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[16]	356	}
	357
	358	#D-#==================================================
	359	#D-function IGCM_sys_Tree
	360	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	361	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	362	#D-
	363	function IGCM_sys_Tree {
[657]	364	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	365	if ( $DEBUG_sys ) ; then
	366	echo "IGCM_sys_Tree :" $@
	367	fi
[16]	368
[1300]	369	\ls -lR ${@}
[16]	370
[657]	371	IGCM_debug_PopStack "IGCM_sys_Tree"
[16]	372	}
	373
	374	#D-#==================================================
	375	#D-function IGCM_sys_Qsub
	376	#D-* Purpose: Qsub new job
	377	#D-* Examples:
	378	#D-
	379	function IGCM_sys_Qsub {
[657]	380	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	381	if ( $DEBUG_sys ) ; then
	382	echo "IGCM_sys_Qsub :" $@
	383	fi
[923]	384	typeset options status
	385	options="-o ${SUBMIT_DIR}/${Script_Output}"
[1179]	386	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	387	status=$?
[949]	388
[1179]	389	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	390	if [ ${status} -gt 0 ] ; then
[1180]	391	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[657]	392	IGCM_debug_Exit "IGCM_sys_Qsub"
[949]	393	else
[1179]	394	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	395	fi
	396	IGCM_debug_PopStack "IGCM_sys_Qsub"
[16]	397	}
	398
	399	#D-#==================================================
	400	#D-function IGCM_sys_QsubPost
	401	#D-* Purpose: Qsub new job on scalaire
	402	#D-* Examples:
	403	#D-
	404	function IGCM_sys_QsubPost {
[657]	405	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	406	if ( $DEBUG_sys ) ; then
	407	echo "IGCM_sys_QsubPost :" $@
	408	fi
[923]	409	typeset options status
	410	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
[1179]	411	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	412	status=$?
[949]	413
[1179]	414	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	415	if [ ${status} -gt 0 ] ; then
[923]	416	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[657]	417	IGCM_debug_Exit "IGCM_sys_QsubPost"
[949]	418	else
[1179]	419	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	420	fi
	421	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[16]	422	}
	423
	424	#D-*************************
[657]	425	#D- File transfer functions
[16]	426	#D-*************************
	427	#D-
	428
	429	#D-#==================================================
[203]	430	#D-function IGCM_sys_RmRunDir
	431	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	432	#D- scheduler will do the job)
	433	#D-* Examples:
	434	#D-
	435	function IGCM_sys_RmRunDir {
[1057]	436	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[657]	437	if ( $DEBUG_sys ) ; then
	438	echo "IGCM_sys_RmRunDir :" $@
	439	fi
[1601]	440
	441	typeset status
	442
	443	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	444	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	445	status=$?
	446
	447	if [ ${status} -gt 0 ] ; then
	448	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
	449	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	450	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	451	else
	452	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	453	fi
[657]	454	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[203]	455	}
	456
	457	#D-#==================================================
[16]	458	#D-function IGCM_sys_Put_Dir
	459	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	460	#D-* Examples:
	461	#D-
	462	function IGCM_sys_Put_Dir {
[657]	463	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	464	if ( $DEBUG_sys ) ; then
	465	echo "IGCM_sys_Put_Dir :" $@
	466	fi
	467	if [ $DRYRUN = 0 ]; then
	468	if [ ! -d ${1} ] ; then
	469	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	470	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	471	return
[16]	472	fi
	473
[717]	474	typeset status
[16]	475
[717]	476	# Only if we use rsync
	477	#IGCM_sys_TestDirArchive $( dirname $2 )
	478	#
	479	#USUAL WAY
[1179]	480	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	481	status=$?
[16]	482
[717]	483	if [ ${status} -gt 0 ] ; then
[811]	484	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	485	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	486	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[16]	487	else
[1179]	488	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	489	fi
[657]	490	fi
	491	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[16]	492	}
	493
	494	#D-#==================================================
	495	#D-function IGCM_sys_Get_Dir
[717]	496	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[16]	497	#D-* Examples:
	498	#D-
	499	function IGCM_sys_Get_Dir {
[657]	500	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	501	if ( $DEBUG_sys ) ; then
	502	echo "IGCM_sys_Get_Dir :" $@
	503	fi
	504	if [ $DRYRUN = 0 ]; then
[1180]	505	typeset NB_ESSAI DELAI status i
	506	# number of tentative
	507	NB_ESSAI=3
	508	# time delay between tentative
	509	DELAI=2
[16]	510
[1180]	511	#
	512	# USUAL WAY
	513	i=0
	514	while [ $i -lt $NB_ESSAI ] ; do
	515	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	516	status=$?
	517	if [ ${status} -gt 0 ] ; then
	518	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
	519	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	520	sleep $DELAI
	521	else
	522	break
	523	fi
	524	(( i = i + 1 ))
	525	done
[16]	526
[717]	527	if [ ${status} -gt 0 ] ; then
[811]	528	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	529	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	530	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[16]	531	else
[1179]	532	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	533	fi
[657]	534	fi
	535	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[16]	536	}
	537
	538	#D-#==================================================
	539	#D-function IGCM_sys_Put_Rest
[717]	540	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[282]	541	#D- File and target directory must exist.
[16]	542	#D-* Examples:
	543	#D-
	544	function IGCM_sys_Put_Rest {
[657]	545	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	546	if ( $DEBUG_sys ) ; then
	547	echo "IGCM_sys_Put_Rest :" $@
	548	fi
	549	if [ $DRYRUN = 0 ]; then
[1624]	550
[657]	551	if [ ! -f ${1} ] ; then
[1624]	552	# The file is missing
	553	if [ X${3} = X'OPTIONAL' ] ; then
	554	echo "WARINING : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	555	IGCM_debug_Print 1 "IGCM_sys_Put_Rest: Optional restart file ${1} is missing, now continue for the next"
	556	else
	557	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	558	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	559	fi
[717]	560
[1624]	561	else
	562	# The file exist
	563
[717]	564	typeset status
	565	#
[1178]	566	# USUAL WAY
[1179]	567	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	568	status=$?
[16]	569
[717]	570	# #RSYNC WITH NETWORK SSH CALL
[1179]	571	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	572	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	573
[717]	574	# #RSYNC WITH NFS USE
[1179]	575	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	576	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	577
[717]	578	# status=$?
	579	# IGCM_sys_Rsync_out $status
	580
[1179]	581	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	582	# (( status=status+$? ))
	583
	584	if [ ${status} -gt 0 ] ; then
[1180]	585	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[1178]	586	[ -f ${1} ] && ls -l ${1}
	587	[ -f ${2} ] && ls -l ${2}
	588	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	589	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	590	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	591	else
[1178]	592
	593	if [ X${JobType} = XRUN ] ; then
	594	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	595	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	596	fi
	597
[1179]	598	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	599	fi
[1624]	600	fi
[657]	601	fi
	602	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
	603	}
[16]	604
[657]	605	#D-#==================================================
[16]	606	#D-function IGCM_sys_Put_Out
[717]	607	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[16]	608	#D-* Examples:
	609	#D-
	610	function IGCM_sys_Put_Out {
[657]	611	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	612	if ( $DEBUG_sys ) ; then
	613	echo "IGCM_sys_Put_Out :" $@
	614	fi
[811]	615
	616	typeset status
	617
[657]	618	if [ $DRYRUN = 0 ]; then
[811]	619	if [ ! -f ${1} ] ; then
[657]	620	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	621	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	622	return 1
[16]	623	fi
[719]	624	#
[811]	625	IGCM_sys_MkdirArchive $( dirname $2 )
	626	#
[657]	627	if [ X${JobType} = XRUN ] ; then
	628	if [ X${3} = X ] ; then
	629	IGCM_sys_Chmod 444 ${1}
	630	fi
	631	fi
[717]	632	#
[16]	633
[1179]	634	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	635	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	636	status=$?
	637	IGCM_sys_Rsync_out $status
[16]	638
[1179]	639	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	640	(( status=status+$? ))
[16]	641
[717]	642	if [ ${status} -gt 0 ] ; then
[811]	643	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
[1179]	644	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	645	IGCM_debug_Exit "IGCM_sys_Put_Out"
[16]	646	else
[1179]	647	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	648	fi
[657]	649	fi
	650	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	651	return 0
[16]	652	}
	653
	654	#D-#==================================================
	655	#D-function IGCM_sys_Get
	656	#D-* Purpose: Get a file from ${ARCHIVE}
	657	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	658	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	659	function IGCM_sys_Get {
[657]	660	IGCM_debug_PushStack "IGCM_sys_Get" $@
[16]	661
[811]	662	typeset DEST status dm_liste
[16]	663
[657]	664	if ( $DEBUG_sys ) ; then
	665	echo "IGCM_sys_Get :" $@
	666	fi
	667	if [ $DRYRUN -le 2 ]; then
	668	if [ X${1} = X'/l' ] ; then
	669	eval set +A dm_liste \${${2}}
	670	else
	671	dm_liste=${1}
[16]	672	fi
[657]	673	eval DEST=\${${#}}
[16]	674
[717]	675	# test if the (first) file is present in the old computation :
[657]	676	IGCM_sys_TestFileArchive ${dm_liste[0]}
[717]	677	status=$?
	678	if [ ${status} -gt 0 ] ; then
[657]	679	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
	680	IGCM_debug_Exit "IGCM_sys_Get"
[811]	681	return
[657]	682	fi
[16]	683
[811]	684	#USUAL WAY
[1179]	685	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	686	status=$?
[16]	687
[811]	688	if [ ${status} -gt 0 ] ; then
	689	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
[1179]	690	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	691	IGCM_debug_Exit "IGCM_sys_Get"
[16]	692	else
[1179]	693	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	694	fi
[657]	695	fi
	696	IGCM_debug_PopStack "IGCM_sys_Get"
[16]	697	}
	698
	699	#D-#==================================================
[1131]	700	#D-function IGCM_sys_GetDate_Monitoring
	701	#D-* Purpose: get the last year for which the monitoring has been computed
	702	#D-* Examples:
	703	#D-
	704	function IGCM_sys_GetDate_Monitoring {
	705	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	706	if ( $DEBUG_sys ) ; then
	707	echo "IGCM_sys_GetDate_Monitoring :" $@
	708	fi
	709
	710	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	711
	712	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	713	}
	714
	715	#D-#==================================================
[343]	716	#D-function IGCM_sys_Dods_Rm
[717]	717	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
[343]	718	#D-* Examples:
	719	#D-
	720	function IGCM_sys_Dods_Rm {
[657]	721	if ( $DEBUG_sys ) ; then
	722	echo "IGCM_sys_Dods_Rm :" $@
	723	fi
	724	return 0
[343]	725	}
	726
	727	#D-#==================================================
	728	#D-function IGCM_sys_Dods_Cp
[717]	729	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
[343]	730	#D-* Examples:
	731	#D-
	732	function IGCM_sys_Dods_Cp {
[657]	733	if ( $DEBUG_sys ) ; then
	734	echo "IGCM_sys_Dods_Cp :" $@
	735	fi
	736	return 0
[343]	737	}
	738
	739	#D-#==================================================
[16]	740	#D-function IGCM_sys_Put_Dods
[811]	741	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
[16]	742	#D-* Examples:
	743	#D-
	744	function IGCM_sys_Put_Dods {
[657]	745	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	746	if ( $DEBUG_sys ) ; then
	747	echo "IGCM_sys_Put_Dods :" $@
	748	fi
	749	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[16]	750	}
	751
[657]	752	##############################################################
	753	# REBUILD OPERATOR
[536]	754
[811]	755	#D-#==================================================
[1163]	756	#D-function IGCM_sys_sync
[1300]	757	#D-* Purpose: flush buffer on disk (dummy function on Obelix)
[1163]	758	#D-* Examples:
	759	#D-
	760	function IGCM_sys_sync {
	761	IGCM_debug_PushStack "IGCM_sys_sync" $@
	762	if ( $DEBUG_sys ) ; then
	763	echo "IGCM_sys_sync :" $@
	764	echo "Dummy call, let the system do that."
	765	fi
	766	IGCM_debug_PopStack "IGCM_sys_sync"
	767	}
	768
[657]	769	############################################################
	770	# Activate Running Environnment Variables
[536]	771
[811]	772	#D-#==================================================
[1239]	773	#D-function IGCM_sys_activ_variables
[811]	774	#D-* Purpose: set environement variables prior to execution
	775	#D-* Examples:
	776	#D-
[657]	777	function IGCM_sys_activ_variables {
	778	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	779	if ( $DEBUG_sys ) ; then
	780	echo "IGCM_sys_activ_variables"
	781	fi
[811]	782
	783	# --------------------------------------------------------------------
[1485]	784	#D- MODULE specifications
	785	# --------------------------------------------------------------------
	786	# Source the file EnvFile containing all module load needed to run the model.
	787	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	788	# it will be used. Otherwise a default file will be used.
	789	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	790	# EnvFile is set in config.card and the file exists
	791	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	792	EnvFile=${config_UserChoices_EnvFile}
	793	else
	794	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
	795	EnvFile=/home/orchideeshare/igcmg/MachineEnvironment/obelix/env_atlas_obelix
	796	fi
	797	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	798	IGCM_debug_Print 1 ${EnvFile}
	799	. ${EnvFile}
	800
	801	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for obelix. "
	802	module list
	803
	804	# --------------------------------------------------------------------
[811]	805	#D- MPI specifications
	806	# --------------------------------------------------------------------
	807
	808	# --------------------------------------------------------------------
	809	#D- Other specifications
	810	# --------------------------------------------------------------------
	811
[1300]	812	ulimit -s unlimited
	813
[657]	814	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[16]	815	}
	816
[657]	817	############################################################
	818	# Desactivate Running Environnment Variables
[16]	819
[811]	820	#D-#==================================================
	821	#D-function IGCM_sys_desactiv_variables
	822	#D-* Purpose: unset environement variables after execution
	823	#D-* Examples:
	824	#D-
[657]	825	function IGCM_sys_desactiv_variables {
	826	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	827	if ( $DEBUG_sys ) ; then
	828	echo "IGCM_sys_desactiv_variables"
	829	fi
[811]	830	# --------------------------------------------------------------------
	831	#D- MPI specifications
	832	# --------------------------------------------------------------------
	833
	834	# --------------------------------------------------------------------
	835	#D- Other specifications
	836	# --------------------------------------------------------------------
	837
[657]	838	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
	839	}
	840
	841	############################################################
[1239]	842	# Update job headers to be used by the scheduler
	843
	844	#D-#==================================================
	845	#D-function IGCM_sys_updateHeaders
	846	#D-* Purpose: Update job headers to be used by the scheduler
	847	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	848	#D-
	849	function IGCM_sys_updateHeaders {
	850	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	851	if ( $DEBUG_sys ) ; then
	852	echo "IGCM_sys_updateHeaders"
	853	fi
	854	typeset file
	855	file=$1
	856
	857	if [ ${executionType} -eq 1 ] ; then
	858	# MPMD + MPI
	859	sed -e "/::openMPthreads::/d" \
	860	-e "s/::JobNumProcTot::/${coreNumber}/" \
	861	${file} > ${file}.tmp
	862
	863	elif [ ${executionType} -eq 2 ] ; then
	864	# MPMD + MPI + OMP
	865	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	866	-e "s/::JobNumProcTot::/${coreNumber}/" \
	867	${file} > ${file}.tmp
	868
	869	elif [ ${executionType} -eq 3 ] ; then
	870	# SPMD + MPI/OMP
	871	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	872	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	873	${file} > ${file}.tmp
	874
	875	elif [ ${executionType} -eq 4 ] ; then
	876	# SPMD + MPI only
	877	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	878	-e "/::openMPthreads::/d" \
	879	${file} > ${file}.tmp
	880
	881	elif [ ${executionType} -eq 5 ] ; then
	882	# SPMD + OMP only
	883	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	884	-e "/::JobNumProcTot::/d" \
	885	${file} > ${file}.tmp
	886
	887	elif [ ${executionType} -eq 6 ] ; then
	888	# SEQUENTIAL THEN
	889	sed -e "s/::JobNumProcTot::/1/" \
	890	-e "/::openMPthreads::/d" \
	891	${file} > ${file}.tmp
	892
	893	fi
	894
	895	IGCM_sys_Mv ${file}.tmp ${file}
	896
	897	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	898	}
	899
	900	############################################################
[800]	901	# Build MPI/OMP scripts run file (dummy function)
[657]	902
[811]	903	#D-#==================================================
	904	#D-function IGCM_sys_build_run_file
	905	#D-* Purpose: build run file (deprecated)
	906	#D-* Examples:
	907	#D-
[657]	908	function IGCM_sys_build_run_file {
[664]	909	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[657]	910	}
	911
	912	############################################################
	913	# Build MPI/OMP scripts
[811]	914
	915	#D-#==================================================
	916	#D-function IGCM_sys_build_execution_scripts
	917	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	918	#D-* Examples:
	919	#D-
[657]	920	function IGCM_sys_build_execution_scripts
	921	{
	922	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	923	if ( $DEBUG_sys ) ; then
	924	echo "IGCM_sys_build_execution_scripts " $@
	925	fi
[1180]	926
[1300]	927	EXECUTION=${HOST_MPIRUN_COMMAND}
[657]	928
[1300]	929	# MPMD mode
[657]	930	if ( ${OK_PARA_MPMD} ) ; then
	931
[1300]	932	# Only MPI (MPMD)
	933	if ( ! ${OK_PARA_OMP} ) ; then
[657]	934
[1300]	935	if [ -f run_file ] ; then
	936	IGCM_sys_Rm -f run_file
	937	fi
	938	touch run_file
[657]	939
[1300]	940	# Build run_file
	941
	942	# First loop on the components for the coupler ie oasis (only if oasis3)
	943	# the coupler ie oasis3 must be the first one
	944	for comp in ${config_ListOfComponents[*]} ; do
	945
	946	eval ExeNameIn=\${config_Executable_${comp}[0]}
	947	eval ExeNameOut=\${config_Executable_${comp}[1]}
	948
	949	# for CPL component only
	950	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
	951	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	952	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	953	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
	954	fi
	955	done
	956
	957	# Then second loop on the components
	958	for comp in ${config_ListOfComponents[*]} ; do
	959
	960	eval ExeNameIn=\${config_Executable_${comp}[0]}
	961	eval ExeNameOut=\${config_Executable_${comp}[1]}
	962
	963	# Only if we really have an executable for the component and not the coupler ie oasis:
	964	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	965
	966	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	967	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	968	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	969	fi
	970	done
	971
	972	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
	973
	974	IGCM_sys_Chmod u+x run_file
	975	if ( $DEBUG_sys ) ; then
	976	echo "run_file contains : "
	977	cat run_file
	978	fi
	979
	980	# MPI-OpenMP (MPMD)
	981	else
	982
[657]	983	# Hosts treatment
	984	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
	985
	986	i=0
	987	rm -f hosts
[1300]	988	IGCM_debug_Print 1 "sys Obelix, Hosts available :"
[657]	989	for nodes in `cat hosts.tmp` ; do
	990	host[$i]=$nodes
	991	echo "${host[$i]} slots=1 max_slots=1" >> hosts
	992	IGCM_debug_Print 1 ${host[$i]}
	993	i=$((i+1))
	994	done
	995	rm -f hosts.tmp
	996
	997	listnodes=${host[*]}
	998
	999	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
	1000
	1001	# Initialisation
[1300]	1002	rank=0
	1003	current_core=0
	1004	core_per_node=6
[657]	1005	init_exec=n
	1006
[1300]	1007	# Loop on the components
[657]	1008	for comp in ${config_ListOfComponents[*]} ; do
	1009
[1300]	1010	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1011	eval ExeNameOut=\${config_Executable_${comp}[1]}
[657]	1012
[1300]	1013	# Not possible if oasis has an executable (i.e old version of oasis3)
	1014	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1015	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1016	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1017	IGCM_debug_Verif_Exit
[657]	1018	fi
	1019
	1020	# Only if we really have an executable for the component :
[1300]	1021	if [ "X${ExeNameOut}" != X\"\" ] ; then
[657]	1022
	1023	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1024	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1025
	1026	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1027	echo "" >> script_${ExeNameOut}.ksh
[1300]	1028	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1029
	1030	# Check if the number of threads is correct
	1031	case ${comp_proc_omp_loc} in
	1032	2\|4\|6)
	1033	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1034	;;
	1035	*)
	1036	IGCM_debug_Exit "ERROR with OMP parameters !"
	1037	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1038	IGCM_debug_Print 2 "Only 2,4,6 as number of OMP threads are possible "
	1039	IGCM_debug_Verif_Exit
	1040	;;
	1041	esac
	1042	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1043	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1044	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1045	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1046	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1047	fi
[1318]	1048
[1317]	1049	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1318]	1050	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	1051	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	1052	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
[657]	1053	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1054
[1300]	1055	if [ ${init_exec} = y ] ; then
	1056	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1057	else
	1058	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1059	init_exec=y
	1060	fi
[657]	1061
[1300]	1062	# Build rankfile : method used to assign cores and nodes for the MPI process
	1063	# Ex :
	1064	#rank 0=curie5296 slot=0,1,2,3
	1065	#rank 1=curie5296 slot=4,5,6,7
	1066	# Example of final command :
	1067	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1068	# with script_lmdz.x.ksh :
	1069	# #!/bin/ksh
	1070	#export OMP_STACKSIZE=200M
	1071	#export OMP_NUM_THREADS=4
	1072	#./lmdz.x
[657]	1073
[1300]	1074	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	1075	(( index_host = current_core / core_per_node ))
	1076	host_value=${host[${index_host}]}
	1077	(( slot = current_core % core_per_node ))
	1078	virg=","
	1079	string_final=""
	1080	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1081	string=$index$virg
	1082	string_final=$string_final$string
	1083	done
	1084	string_final=$( echo $string_final \| sed "s/.$//" )
	1085	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1086	(( rank = rank + 1 ))
	1087	(( current_core = current_core + comp_proc_omp_loc ))
[657]	1088	done
	1089	fi
	1090	done
[16]	1091	fi
	1092
[1300]	1093	# Only one executable (SPMD mode).
	1094	else
[657]	1095
[664]	1096	for comp in ${config_ListOfComponents[*]} ; do
[657]	1097
[664]	1098	# Only if we really have an executable for the component :
	1099	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1100	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[657]	1101
[664]	1102	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1103	echo "" >> script_${ExeNameOut}.ksh
	1104	if ( ${OK_PARA_OMP} ) ; then
	1105	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1300]	1106	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1107	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[664]	1108	fi
[1480]	1109
	1110	if [ ${mpiTasks} -eq 1 ] && [ ${openMPthreads} -eq 1 ] ; then
	1111	# Case with only one proc
	1112	# mpirun should not be used at obelix for only one proc
	1113	# Output text per process
	1114	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1115	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1116	EXECUTION="time ./script_${ExeNameOut}.ksh"
	1117	elif ( ${OK_PARA_MPI} ) ; then
[664]	1118	# Default : mpirun used if nb_proc gt 1
[1480]	1119	# Output text per process
[1317]	1120	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
[664]	1121	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1122	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
[1481]	1123	elif [ ${executionType} -eq 6 ] ; then
	1124	# Sequentiel case: this is the same as the first case.
	1125	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1126	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1127	EXECUTION="time ./script_${ExeNameOut}.ksh"
[664]	1128	else
[1480]	1129	# Strange case.
	1130	IGCM_debug_Exit "ERROR with OK_PARA_MPI"
	1131	IGCM_debug_Print 2 "sys Obelix : OK_PARA_MPI=False and mpiTask/openMPthreds different from 1"
	1132	IGCM_debug_Verif_Exit
[664]	1133	fi
[1002]	1134
	1135	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
	1136	cat script_${ExeNameOut}.ksh
	1137
[664]	1138	fi
[1180]	1139
[664]	1140	done
[657]	1141
	1142	fi
	1143
[811]	1144	IGCM_debug_Print 1 "sys Obelix : execution command is"
	1145	IGCM_debug_Print 1 "$EXECUTION"
[657]	1146
	1147	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[16]	1148	}
	1149
[933]	1150	#D-#==================================================
	1151	#D-function IGCM_sys_check_path
	1152	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1153	#D-* do not point to an important use directory. Stop immediately in that case.
	1154	#D-* Examples:
	1155	#D-
	1156	function IGCM_sys_check_path {
	1157	IGCM_debug_PushStack "IGCM_sys_check_path"
	1158	if ( $DEBUG_sys ) ; then
	1159	echo "IGCM_sys_check_path"
	1160	fi
[811]	1161
[933]	1162	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
	1163	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1164	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1165	IGCM_debug_Exit "This will stop the job"
[933]	1166	fi
	1167	IGCM_debug_PopStack "IGCM_sys_check_path"
	1168	}
	1169
[811]	1170	#D-#==================================================
	1171	#D-function IGCM_sys_check_quota. Dummy call here
	1172	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1173	#D-* Examples:
	1174	#D-
[657]	1175	function IGCM_sys_check_quota {
	1176	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1177	if ( $DEBUG_sys ) ; then
	1178	echo "IGCM_sys_check_quota"
	1179	fi
[1300]	1180
	1181	# TO BE IMPLEMENTED
	1182
[657]	1183	IGCM_debug_PopStack "IGCM_sys_check_quota"
	1184	}
	1185
[811]	1186	#D-#==================================================
[1286]	1187	#D-function IGCM_sys_projectAccounting
	1188	#D-* Purpose: store project accounting information in a file
	1189	#D-* Examples:
	1190	#D-
	1191	function IGCM_sys_projectAccounting {
	1192	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1193	if ( $DEBUG_sys ) ; then
	1194	echo "IGCM_sys_check_quota"
	1195	fi
	1196
	1197	touch $1
	1198
	1199	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1200	}
	1201
	1202	#D-#==================================================
[1244]	1203	#D-function IGCM_sys_getJobSchedulerID
	1204	#D-* Purpose: Get the job ID during execution
	1205	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1206	#D-
	1207	function IGCM_sys_getJobSchedulerID {
	1208	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1209	if ( $DEBUG_sys ) ; then
	1210	echo "IGCM_sys_getJobSchedulerID"
	1211	fi
	1212
	1213	eval ${1}=$( echo ${PBS_JOBID} \| awk -F. '{print $1}' )
	1214
	1215	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1216	}
	1217
	1218	#D-#==================================================
[1124]	1219	#D-function IGCM_sys_GetJobID
[1244]	1220	#D-* Purpose: Get the job ID from the JobName
[1124]	1221	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1222	#D-
	1223	function IGCM_sys_GetJobID {
	1224	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1225	if ( $DEBUG_sys ) ; then
	1226	echo "IGCM_sys_GetJobID"
	1227	fi
	1228
	1229	# With -f option, the full job name is given in the last column
[1300]	1230	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
[1124]	1231
	1232	eval ${3}=${ID}
	1233	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1234	}
	1235
	1236	#D-#==================================================
[811]	1237	#D-function IGCM_sys_CountJobInQueue
[1244]	1238	#D-* Purpose: Count number of users job
[811]	1239	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1240	#D-
	1241	function IGCM_sys_CountJobInQueue {
	1242	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1243	if ( $DEBUG_sys ) ; then
	1244	echo "IGCM_sys_CountJobInQueue"
	1245	fi
	1246	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1247	}
[1272]	1248
	1249	#D-#==================================================
	1250	#D-function IGCM_sys_atlas
	1251	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1252	#D-* Examples:
	1253	#D-
	1254	function IGCM_sys_atlas {
	1255	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1256	if ( $DEBUG_sys ) ; then
	1257	echo "IGCM_sys_atlas :" $@
	1258	fi
	1259
	1260	typeset status
	1261
	1262	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1263	status=$?
	1264	if [ ${status} -gt 0 ] ; then
	1265	echo "IGCM_sys_atlas : error code ${status}"
	1266	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1267	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1268	IGCM_debug_PopStack "IGCM_sys_atlas"
	1269	return 1
	1270	else
	1271	IGCM_debug_PopStack "IGCM_sys_atlas"
	1272	return 0
	1273	fi
	1274
	1275	IGCM_debug_PopStack "IGCM_sys_atlas"
	1276	}
	1277
[1380]	1278
	1279	#D-#==================================================
	1280	#D-function IGCM_sys_rebuild_nemo
	1281	#D-* Purpose: rebuild nemo parallel files with general rebuild
	1282	#D-* Examples:
	1283	#D-
	1284
	1285	function IGCM_sys_rebuild_nemo {
	1286	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1287	if ( $DEBUG_sys ) ; then
	1288	echo "IGCM_sys_rebuild_nemo :" $@
	1289	fi
	1290
	1291	nemo_generic_restart_file_name_out=${1}
	1292	nemo_extension_out=${3}
	1293	shift ; shift ; shift
	1294
	1295	IGCM_sys_rebuild -o ${nemo_generic_restart_file_name_out}.${nemo_extension_out} $@
	1296
	1297	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1298	}
	1299

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh

Download in other formats: