Context Navigation

libIGCM_sys_default.ksh @ 1624

Last change on this file since 1624 was 1624, checked in by jgipsl, 4 months ago

Added possibility to declare a restart file as optional which means that libIGCM want stop if the file is not found. Only a warning will be written.

For example, in the following case, if the routing_restart file is not available when starting the model or if it is not produced, libIGCM will not stop. But if the file sechiba_rest is not available, libIGCM will stop as usual.

[RestartFiles]
List=   (sechiba_rest_out.nc, sechiba_rest.nc, sechiba_rest_in.nc) ,\
        (routing_restart.nc, routing_restart.nc, routing_start.nc, OPTIONAL)

Read optional argument OPTIONAL on the line where the restart file is declared.
Each time when a restart file is not found for coping in or out, if the argument OPTIONAL is set, then just continue. If not set, the file is considered as MANDATORY and libIGCM will exist if it is missing as before.
on irene-amd and jean-zay, corrected a bug for the case in config.card DataProject?=DEFAULT

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 39.5 KB

Rev	Line
[664]	1	#!/bin/ksh
[2]	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip
[373]	5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
[2]	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[664]	16	# if you use the prefix #D- for comments to be extracted.
[2]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
[664]	19
[2]	20	#D-#==================================================
	21	#D-LibIGCM_sys Default host
	22	#D-#==================================================
	23	#D-
[664]	24	#D- This ksh library if a layer under some usefull
[800]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
[2]	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
[117]	34	#====================================================
[800]	35	# set DEBUG_sys to true to output calls of function
[2]	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
[117]	38	#====================================================
[2]	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[2]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
[800]	58	# Global Variables :
[2]	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
[1300]	64	# Host user names project maxCpuTime
[217]	65	# $hostname ou hostname
[1180]	66	typeset HOST=${HOST:=$( hostname )}
[2]	67	# $username ou whoami
[1180]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[71]	69	# $hostname of the MASTER job
[1180]	70	typeset MASTER=${MASTER:=$( hostname )}
[1201]	71	# project name
	72	typeset PROJECT=NONE
	73	# jobWarningDelay in seconds
	74	typeset jobWarningDelay=NONE
[2]	75
	76	#D-
	77	#D-#==================================================
	78	#D-Program used in libIGCM
	79	#D-#==================================================
	80
[742]	81	# Submit command
	82	typeset SUBMIT=${SUBMIT:=qsub}
[2]	83	# rsync with path
[664]	84	typeset -r RSYNC=/usr/bin/rsync
[800]	85	# RSYNC_opt args to rsync
[117]	86	typeset -r RSYNC_opt="-va"
[664]	87	# ie storage filesystem
	88	typeset -r STOREHOST=${MASTER}
[305]	89
[2]	90	#====================================================
[1180]	91	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[305]	92	#====================================================
	93	#. /home/${LOGIN}/.atlas_env_${HOST}_ksh
[1180]	94	#export PATH=${PATH}:/home/${LOGIN}/rebuild/bin/
[1485]	95	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
[2]	96
[117]	97	#====================================================
[2]	98	# Host specific DIRECTORIES
	99	#====================================================
	100
	101	#====================================================
[664]	102	#- MirrorlibIGCM for frontend
	103	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
[2]	104
	105	#====================================================
[800]	106	#- libIGCM_POST for frontend
[343]	107	typeset -r libIGCM_POST=${libIGCM}
	108
	109	#====================================================
[664]	110	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	111	typeset -r R_EXE="${MODIPSL}/bin"
	112
	113	#====================================================
[2]	114	#- SUBMIT_DIR : submission dir
[1124]	115	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
[2]	116
	117	#====================================================
[725]	118	#- IN
[1484]	119	typeset -r R_IN=${R_IN:=/home_local/${LOGIN}/IGCM}
[725]	120
	121	#====================================================
[2]	122	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1484]	123	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home_local/${LOGIN}/RUN_DIR/tmp$$}
[2]	124
	125	#====================================================
[1179]	126	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	127	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	128
	129	#====================================================
[2]	130	#- HOST_MPIRUN_COMMAND
	131	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
	132
[61]	133	#====================================================
	134	#- Max number of arguments passed to nco operator or demigration command
	135	UNIX_MAX_LIMIT=360
	136
[664]	137	#====================================================
[1300]	138	#- set PackDefault to false on the default machine
[1180]	139	PackDefault=false
	140
	141	#====================================================
[664]	142	#- Default number of MPI task for IPSL coupled model
	143	#- required for backward compatibility
	144	#-
[1300]	145	DEFAULT_NUM_PROC_OCE=5
[664]	146	DEFAULT_NUM_PROC_CPL=1
	147	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	148
[2]	149	#D-#==================================================
[1057]	150	#D-function IGCM_sys_defineArchives
	151	#D-* Purpose:
	152	#D-* Define ARCHIVE : Dedicated to large files
	153	#D-* Define STORAGE : Dedicated to small/medium files
	154	#D-* Define R_OUT : Output tree located on ARCHIVE
	155	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	156	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
	157	#D-* if SpaceName=TEST nothing special will hapen
[725]	158	#D-* Examples:
	159	#D-
[1057]	160	function IGCM_sys_defineArchives {
	161	IGCM_debug_PushStack "IGCM_sys_defineArchives"
	162
[1060]	163	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	164	#====================================================
	165	#- ARCHIVE (dedicated to large files)
[1063]	166	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	167	else
	168	#====================================================
	169	#- ARCHIVE (dedicated to large files)
[1484]	170	ARCHIVE=${ARCHIVE:=/home_local/${LOGIN}}
[1057]	171	fi
	172
[1060]	173	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	174	#====================================================
	175	#- STORAGE (dedicated to small/medium files)
[1063]	176	STORAGE=${config_UserChoices_STORAGE}
[1057]	177	else
	178	#====================================================
	179	#- STORAGE (dedicated to small/medium files)
[1063]	180	STORAGE=${ARCHIVE}
[1057]	181	fi
	182
[1178]	183	# ON DEFAULT MACHINE NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
[1057]	184
	185	#====================================================
	186	#- R_OUT
[1063]	187	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	188
	189	#====================================================
	190	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	191	R_FIG=${STORAGE}/IGCM_OUT
[1057]	192
	193	#====================================================
[1431]	194	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	195	CMIP6_BUF=${STORAGE}/IGCM_OUT
	196
	197	#====================================================
[1057]	198	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	199	R_BUF=${STORAGE}/IGCM_OUT
[1057]	200
	201	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	202	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	203	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	204
	205	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	206	}
	207
	208	#D-#==================================================
[37]	209	#D-function IGCM_sys_RshArchive
	210	#D-* Purpose: Archive rsh command
	211	#D-* Examples:
	212	#D-
	213	function IGCM_sys_RshArchive {
[664]	214	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
[800]	215	/bin/ksh <<-EOF
[40]	216	${@}
	217	EOF
[811]	218	status=$?
	219	if [ ${status} -gt 0 ] ; then
	220	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[664]	221	IGCM_debug_Exit "IGCM_sys_RshArchive"
	222	fi
	223	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[37]	224	}
	225
	226	#D-#==================================================
[1178]	227	#D-function IGCM_sys_RshArchive_NoError
	228	#D-* Purpose: Archive rsh command, without error
	229	#D-* used only in monitoring.job
	230	#D-* Examples:
	231	#D-
	232	function IGCM_sys_RshArchive_NoError {
	233	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	234	/bin/ksh <<-EOF
[1181]	235	${@} 2> /dev/null
[1178]	236	EOF
	237	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	238	}
	239
	240	#D-#==================================================
[2]	241	#D-function IGCM_sys_MkdirArchive
	242	#D-* Purpose: Mkdir on Archive
	243	#D-* Examples:
	244	#D-
	245	function IGCM_sys_MkdirArchive {
[664]	246	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	247	if ( $DEBUG_sys ) ; then
	248	echo "IGCM_sys_MkdirArchive :" $@
	249	fi
[800]	250	#- creation de repertoire sur le serveur fichier
[1178]	251	if [ ! -d ${1} ]; then
	252	\mkdir -p $1
	253	status=$?
[811]	254
[1178]	255	if [ ${status} -gt 0 ] ; then
	256	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
	257	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
	258	fi
[664]	259	fi
	260	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[2]	261	}
	262
	263	#D-#==================================================
	264	#D-function IGCM_sys_TestDirArchive
	265	#D-* Purpose: Test Directory that must exists on Archive
	266	#D-* Examples:
	267	#D-
	268	function IGCM_sys_TestDirArchive {
[664]	269	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	270	if ( $DEBUG_sys ) ; then
	271	echo "IGCM_sys_TestDirArchive :" $@
	272	fi
	273	typeset ExistFlag
	274	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	275	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	276	return ${ExistFlag}
[2]	277	}
	278
	279	#D-#==================================================
[800]	280	#D-function IGCM_sys_IsFileArchived
[1180]	281	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[800]	282	#D-* Examples:
	283	#D-
	284	function IGCM_sys_IsFileArchived {
	285	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	286	if ( $DEBUG_sys ) ; then
	287	echo "IGCM_sys_IsFileArchived :" $@
	288	fi
	289	typeset IsArchivedFlag
	290	# Never archived for this system
	291	IsArchivedFlag=1
	292	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	293
	294	return ${IsArchivedFlag}
	295	}
	296
	297	#D-#==================================================
[2]	298	#D-function IGCM_sys_TestFileArchive
	299	#D-* Purpose: Test file that must NOT EXISTS on Archive
	300	#D-* Examples:
	301	#D-
	302	function IGCM_sys_TestFileArchive {
[664]	303	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	304	typeset ExistFlag
	305	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	306	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[67]	307
[664]	308	return ${ExistFlag}
[67]	309	}
	310
	311	#D-#==================================================
	312	#D-function IGCM_sys_CountFileArchive
	313	#D-* Purpose: Count files on Archive filesystem
	314	#D-* Examples:
	315	#D-
	316	function IGCM_sys_CountFileArchive {
[664]	317	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	318	ls ${@} 2>/dev/null \| wc -l
	319	if [ $? -gt 0 ] ; then
	320	echo "IGCM_sys_CountFileArchive : erreur."
	321	fi
	322	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[2]	323	}
	324
	325	#D-#==================================================
	326	#D-function IGCM_sys_Tree
	327	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	328	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	329	#D-
	330	function IGCM_sys_Tree {
[664]	331	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	332	if ( $DEBUG_sys ) ; then
	333	echo "IGCM_sys_Tree :" $@
	334	fi
[2]	335
[1300]	336	\ls -lR ${@}
[2]	337
[664]	338	IGCM_debug_PopStack "IGCM_sys_Tree"
[2]	339	}
	340
	341	#D-#==================================================
	342	#D-function IGCM_sys_Qsub
	343	#D-* Purpose: Qsub new job
	344	#D-* Examples:
	345	#D-
	346	function IGCM_sys_Qsub {
[664]	347	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	348	if ( $DEBUG_sys ) ; then
	349	echo "IGCM_sys_Qsub :" $@
	350	fi
[800]	351	echo "${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &"
	352	${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &
[664]	353	if [ $? -gt 0 ] ; then
[800]	354	echo "IGCM_sys_Qsub : erreur ${options} $1"
[664]	355	IGCM_debug_Exit "IGCM_sys_Qsub"
	356	fi
	357	IGCM_debug_PopStack "IGCM_sys_Qsub"
[2]	358	}
	359
	360	#D-#==================================================
	361	#D-function IGCM_sys_QsubPost
	362	#D-* Purpose: Qsub new job on scalaire
	363	#D-* Examples:
	364	#D-
	365	function IGCM_sys_QsubPost {
[664]	366	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	367	if ( $DEBUG_sys ) ; then
	368	echo "IGCM_sys_QsubPost :" $@
	369	fi
	370	echo "${libIGCM_POST}/$1.job > ${POST_DIR}/$1.${PeriodDateEnd}.out 2>&1 &"
	371	${libIGCM_POST}/$1.job > ${POST_DIR}/${Script_Post_Output}.out 2>&1 &
	372	if [ $? -gt 0 ] ; then
	373	echo "IGCM_sys_QsubPost : erreur " $@
	374	IGCM_debug_Exit "IGCM_sys_QsubPost"
	375	fi
	376	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[2]	377	}
	378
	379	#D-*************************
[664]	380	#D- File transfer functions
[2]	381	#D-*************************
	382	#D-
	383
	384	#D-#==================================================
[203]	385	#D-function IGCM_sys_RmRunDir
	386	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	387	#D- scheduler will do the job)
	388	#D-* Examples:
	389	#D-
	390	function IGCM_sys_RmRunDir {
[1057]	391	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[664]	392	if ( $DEBUG_sys ) ; then
	393	echo "IGCM_sys_RmRunDir :" $@
[1183]	394	IGCM_debug_Print 1 "Dummy call, let the user do that."
[664]	395	fi
	396	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[203]	397	}
	398
	399	#D-#==================================================
[2]	400	#D-function IGCM_sys_Put_Dir
	401	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	402	#D-* Examples:
	403	#D-
	404	function IGCM_sys_Put_Dir {
[664]	405	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	406	if ( $DEBUG_sys ) ; then
	407	echo "IGCM_sys_Put_Dir :" $@
	408	fi
	409	if [ $DRYRUN = 0 ]; then
	410	if [ ! -d ${1} ] ; then
	411	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	412	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	413	return
[2]	414	fi
	415
[717]	416	typeset status
[2]	417
[717]	418	# Only if we use rsync
	419	#IGCM_sys_TestDirArchive $( dirname $2 )
	420	#
	421	#USUAL WAY
[1179]	422	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	423	status=$?
[2]	424
[717]	425	if [ ${status} -gt 0 ] ; then
[811]	426	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	427	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	428	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[2]	429	else
[1179]	430	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[2]	431	fi
[664]	432	fi
	433	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[2]	434	}
	435
	436	#D-#==================================================
	437	#D-function IGCM_sys_Get_Dir
[717]	438	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[2]	439	#D-* Examples:
	440	#D-
	441	function IGCM_sys_Get_Dir {
[664]	442	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	443	if ( $DEBUG_sys ) ; then
	444	echo "IGCM_sys_Get_Dir :" $@
	445	fi
	446	if [ $DRYRUN = 0 ]; then
[1180]	447	typeset NB_ESSAI DELAI status i
	448	# number of tentative
	449	NB_ESSAI=3
	450	# time delay between tentative
	451	DELAI=2
[2]	452
[1180]	453	#
	454	# USUAL WAY
	455	i=0
	456	while [ $i -lt $NB_ESSAI ] ; do
	457	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	458	status=$?
	459	if [ ${status} -gt 0 ] ; then
	460	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
	461	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	462	sleep $DELAI
	463	else
	464	break
	465	fi
	466	(( i = i + 1 ))
	467	done
[2]	468
[811]	469	if [ ${status} -gt 0 ] ; then
	470	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	471	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	472	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[2]	473	else
[1179]	474	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[2]	475	fi
[664]	476	fi
	477	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[2]	478	}
	479
	480	#D-#==================================================
	481	#D-function IGCM_sys_Put_Rest
[717]	482	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[117]	483	#D- File and target directory must exist.
[2]	484	#D-* Examples:
	485	#D-
	486	function IGCM_sys_Put_Rest {
[664]	487	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	488	if ( $DEBUG_sys ) ; then
	489	echo "IGCM_sys_Put_Rest :" $@
	490	fi
	491	if [ $DRYRUN = 0 ]; then
	492	if [ ! -f ${1} ] ; then
[1624]	493	# The file is missing
	494	if [ X${3} = X'OPTIONAL' ] ; then
	495	echo "WARINING : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	496	IGCM_debug_Print 1 "IGCM_sys_Put_Rest: Optional restart file ${1} is missing, now continue for the next"
	497	else
	498	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	499	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	500	fi
	501
	502	else
	503	# The file exist
[717]	504	typeset status
	505	#
[1178]	506	# USUAL WAY
[1179]	507	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	508	status=$?
[2]	509
[717]	510	# #RSYNC WITH NETWORK SSH CALL
[1179]	511	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	512	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[2]	513
[717]	514	# #RSYNC WITH NFS USE
[1179]	515	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	516	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[2]	517
[717]	518	# status=$?
	519	# IGCM_sys_Rsync_out $status
	520
[1179]	521	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	522	# (( status=status+$? ))
	523
	524	if [ ${status} -gt 0 ] ; then
[1180]	525	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[1178]	526	[ -f ${1} ] && ls -l ${1}
	527	[ -f ${2} ] && ls -l ${2}
	528	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	529	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	530	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	531	else
[1178]	532
	533	if [ X${JobType} = XRUN ] ; then
	534	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	535	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	536	fi
	537
[1179]	538	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	539	fi
[1624]	540	fi
[664]	541	fi
	542	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
	543	}
[2]	544
[664]	545	#D-#==================================================
[2]	546	#D-function IGCM_sys_Put_Out
[717]	547	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[2]	548	#D-* Examples:
	549	#D-
	550	function IGCM_sys_Put_Out {
[664]	551	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	552	if ( $DEBUG_sys ) ; then
	553	echo "IGCM_sys_Put_Out :" $@
	554	fi
[811]	555
	556	typeset status
	557
[664]	558	if [ $DRYRUN = 0 ]; then
[811]	559	if [ ! -f ${1} ] ; then
[664]	560	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	561	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	562	return 1
[2]	563	fi
[800]	564	#
[811]	565	IGCM_sys_MkdirArchive $( dirname $2 )
	566	#
[664]	567	if [ X${JobType} = XRUN ] ; then
	568	if [ X${3} = X ] ; then
	569	IGCM_sys_Chmod 444 ${1}
	570	fi
	571	fi
[800]	572	#
[2]	573
[1179]	574	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	575	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	576	status=$?
	577	IGCM_sys_Rsync_out $status
[2]	578
[1179]	579	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	580	(( status=status+$? ))
[2]	581
[811]	582	if [ ${status} -gt 0 ] ; then
	583	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
[1179]	584	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	585	IGCM_debug_Exit "IGCM_sys_Put_Out"
[2]	586	else
[1179]	587	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[2]	588	fi
[664]	589	fi
	590	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	591	return 0
[2]	592	}
	593
	594	#D-#==================================================
	595	#D-function IGCM_sys_Get
	596	#D-* Purpose: Get a file from ${ARCHIVE}
	597	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	598	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	599	function IGCM_sys_Get {
[664]	600	IGCM_debug_PushStack "IGCM_sys_Get" $@
[2]	601
[811]	602	typeset DEST status dm_liste
[2]	603
[664]	604	if ( $DEBUG_sys ) ; then
	605	echo "IGCM_sys_Get :" $@
	606	fi
	607	if [ $DRYRUN -le 2 ]; then
	608	if [ X${1} = X'/l' ] ; then
	609	eval set +A dm_liste \${${2}}
	610	else
	611	dm_liste=${1}
[2]	612	fi
[664]	613	eval DEST=\${${#}}
[2]	614
[800]	615	# test if the (first) file is present in the old computation :
[664]	616	IGCM_sys_TestFileArchive ${dm_liste[0]}
[811]	617	status=$?
	618	if [ ${status} -gt 0 ] ; then
[664]	619	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
	620	IGCM_debug_Exit "IGCM_sys_Get"
[811]	621	return
[664]	622	fi
[2]	623
[811]	624	#USUAL WAY
[1179]	625	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	626	status=$?
[2]	627
[811]	628	if [ ${status} -gt 0 ] ; then
	629	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
[1179]	630	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[664]	631	IGCM_debug_Exit "IGCM_sys_Get"
[2]	632	else
[1179]	633	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[2]	634	fi
[664]	635	fi
	636	IGCM_debug_PopStack "IGCM_sys_Get"
[2]	637	}
	638
	639	#D-#==================================================
[1178]	640	#D-function IGCM_sys_GetDate_Monitoring
	641	#D-* Purpose: get the last year for which the monitoring has been computed
	642	#D-* Examples:
	643	#D-
	644	function IGCM_sys_GetDate_Monitoring {
	645	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	646	if ( $DEBUG_sys ) ; then
	647	echo "IGCM_sys_GetDate_Monitoring :" $@
	648	fi
	649
	650	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	651
	652	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	653	}
	654
	655	#D-#==================================================
[343]	656	#D-function IGCM_sys_Dods_Rm
[717]	657	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
[343]	658	#D-* Examples:
	659	#D-
	660	function IGCM_sys_Dods_Rm {
[664]	661	if ( $DEBUG_sys ) ; then
	662	echo "IGCM_sys_Dods_Rm :" $@
	663	fi
	664	return 0
[343]	665	}
	666
	667	#D-#==================================================
	668	#D-function IGCM_sys_Dods_Cp
[811]	669	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
[343]	670	#D-* Examples:
	671	#D-
	672	function IGCM_sys_Dods_Cp {
[664]	673	if ( $DEBUG_sys ) ; then
	674	echo "IGCM_sys_Dods_Cp :" $@
	675	fi
	676	return 0
[343]	677	}
	678
	679	#D-#==================================================
[2]	680	#D-function IGCM_sys_Put_Dods
[811]	681	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
[2]	682	#D-* Examples:
	683	#D-
	684	function IGCM_sys_Put_Dods {
[664]	685	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	686	if ( $DEBUG_sys ) ; then
	687	echo "IGCM_sys_Put_Dods :" $@
	688	fi
	689	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[2]	690	}
	691
[664]	692	##############################################################
	693	# REBUILD OPERATOR
[2]	694
[811]	695	#D-#==================================================
[1163]	696	#D-function IGCM_sys_sync
[1300]	697	#D-* Purpose: flush buffer on disk (dummy function on Default machine)
[1163]	698	#D-* Examples:
	699	#D-
	700	function IGCM_sys_sync {
	701	IGCM_debug_PushStack "IGCM_sys_sync" $@
	702	if ( $DEBUG_sys ) ; then
	703	echo "IGCM_sys_sync :" $@
	704	echo "Dummy call, let the system do that."
	705	fi
	706	IGCM_debug_PopStack "IGCM_sys_sync"
	707	}
	708
[664]	709	############################################################
	710	# Activate Running Environnment Variables
	711
[811]	712	#D-#==================================================
[1239]	713	#D-function IGCM_sys_activ_variables
[811]	714	#D-* Purpose: set environement variables prior to execution
	715	#D-* Examples:
	716	#D-
[664]	717	function IGCM_sys_activ_variables {
	718	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	719	if ( $DEBUG_sys ) ; then
	720	echo "IGCM_sys_activ_variables"
	721	fi
[811]	722
	723	# --------------------------------------------------------------------
[1485]	724	#D- MODULE specifications
	725	# --------------------------------------------------------------------
	726	# Source the file EnvFile containing all module load needed to run the model.
	727	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	728	# it will be used. Otherwise at default host, no file will be used.
	729	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	730	# EnvFile is set in config.card and the file exists
	731	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	732	EnvFile=${config_UserChoices_EnvFile}
	733
	734	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	735	IGCM_debug_Print 1 ${EnvFile}
	736	. ${EnvFile}
	737	else
	738	IGCM_debug_Print 1 "IGCM_sys_active_variables : No modules will be loaded in default set up."
	739	fi
	740
	741	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for default system. "
	742	module list
	743
	744	# --------------------------------------------------------------------
[811]	745	#D- MPI specifications
	746	# --------------------------------------------------------------------
	747
	748	# --------------------------------------------------------------------
	749	#D- Other specifications
	750	# --------------------------------------------------------------------
	751
[1300]	752	ulimit -s unlimited
	753
[664]	754	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[2]	755	}
	756
[664]	757	############################################################
	758	# Desactivate Running Environnment Variables
[2]	759
[811]	760	#D-#==================================================
	761	#D-function IGCM_sys_desactiv_variables
	762	#D-* Purpose: unset environement variables after execution
	763	#D-* Examples:
	764	#D-
[664]	765	function IGCM_sys_desactiv_variables {
	766	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	767	if ( $DEBUG_sys ) ; then
	768	echo "IGCM_sys_desactiv_variables"
	769	fi
[811]	770	# --------------------------------------------------------------------
	771	#D- MPI specifications
	772	# --------------------------------------------------------------------
	773
	774	# --------------------------------------------------------------------
	775	#D- Other specifications
	776	# --------------------------------------------------------------------
	777
[664]	778	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
	779	}
	780
	781	############################################################
[1239]	782	# Update job headers to be used by the scheduler
	783
	784	#D-#==================================================
	785	#D-function IGCM_sys_updateHeaders
	786	#D-* Purpose: Update job headers to be used by the scheduler
	787	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	788	#D-
	789	function IGCM_sys_updateHeaders {
	790	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	791	if ( $DEBUG_sys ) ; then
	792	echo "IGCM_sys_updateHeaders"
	793	fi
	794	typeset file
	795	file=$1
	796
	797	if [ ${executionType} -eq 1 ] ; then
	798	# MPMD + MPI
	799	sed -e "/::openMPthreads::/d" \
	800	-e "s/::JobNumProcTot::/${coreNumber}/" \
	801	${file} > ${file}.tmp
	802
	803	elif [ ${executionType} -eq 2 ] ; then
	804	# MPMD + MPI + OMP
	805	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	806	-e "s/::JobNumProcTot::/${coreNumber}/" \
	807	${file} > ${file}.tmp
	808
	809	elif [ ${executionType} -eq 3 ] ; then
	810	# SPMD + MPI/OMP
	811	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	812	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	813	${file} > ${file}.tmp
	814
	815	elif [ ${executionType} -eq 4 ] ; then
	816	# SPMD + MPI only
	817	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	818	-e "/::openMPthreads::/d" \
	819	${file} > ${file}.tmp
	820
	821	elif [ ${executionType} -eq 5 ] ; then
	822	# SPMD + OMP only
	823	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	824	-e "/::JobNumProcTot::/d" \
	825	${file} > ${file}.tmp
	826
	827	elif [ ${executionType} -eq 6 ] ; then
	828	# SEQUENTIAL THEN
	829	sed -e "s/::JobNumProcTot::/1/" \
	830	-e "/::openMPthreads::/d" \
	831	${file} > ${file}.tmp
	832
	833	fi
	834
	835	IGCM_sys_Mv ${file}.tmp ${file}
	836
	837	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	838	}
	839
	840	############################################################
[800]	841	# Build MPI/OMP scripts run file (dummy function)
[664]	842
[811]	843	#D-#==================================================
	844	#D-function IGCM_sys_build_run_file
	845	#D-* Purpose: build run file (deprecated)
	846	#D-* Examples:
	847	#D-
[664]	848	function IGCM_sys_build_run_file {
	849	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
	850	}
	851
	852	############################################################
	853	# Build MPI/OMP scripts
[811]	854
	855	#D-#==================================================
	856	#D-function IGCM_sys_build_execution_scripts
	857	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	858	#D-* Examples:
	859	#D-
[664]	860	function IGCM_sys_build_execution_scripts
	861	{
	862	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	863	if ( $DEBUG_sys ) ; then
	864	echo "IGCM_sys_build_execution_scripts " $@
	865	fi
[1180]	866
[1300]	867	EXECUTION=${HOST_MPIRUN_COMMAND}
[664]	868
[1300]	869	# MPMD mode
[664]	870	if ( ${OK_PARA_MPMD} ) ; then
	871
[1300]	872	# Only MPI (MPMD)
	873	if ( ! ${OK_PARA_OMP} ) ; then
[664]	874
[1300]	875	if [ -f run_file ] ; then
	876	IGCM_sys_Rm -f run_file
	877	fi
	878	touch run_file
[664]	879
[1300]	880	# Build run_file
	881
	882	# First loop on the components for the coupler ie oasis (only if oasis3)
	883	# the coupler ie oasis3 must be the first one
	884	for comp in ${config_ListOfComponents[*]} ; do
	885
	886	eval ExeNameIn=\${config_Executable_${comp}[0]}
	887	eval ExeNameOut=\${config_Executable_${comp}[1]}
	888
	889	# for CPL component only
	890	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
	891	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	892	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	893	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
	894	fi
	895	done
	896
	897	# Then second loop on the components
	898	for comp in ${config_ListOfComponents[*]} ; do
	899
	900	eval ExeNameIn=\${config_Executable_${comp}[0]}
	901	eval ExeNameOut=\${config_Executable_${comp}[1]}
	902
	903	# Only if we really have an executable for the component and not the coupler ie oasis:
	904	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	905
	906	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	907	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	908	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	909	fi
	910	done
	911
	912	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
	913
	914	IGCM_sys_Chmod u+x run_file
	915	if ( $DEBUG_sys ) ; then
	916	echo "run_file contains : "
	917	cat run_file
	918	fi
	919
	920	# MPI-OpenMP (MPMD)
	921	else
	922
[664]	923	# Hosts treatment
	924	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
	925
	926	i=0
	927	rm -f hosts
[1300]	928	IGCM_debug_Print 1 "sys $( hostname ), Hosts available :"
[664]	929	for nodes in `cat hosts.tmp` ; do
	930	host[$i]=$nodes
	931	echo "${host[$i]} slots=1 max_slots=1" >> hosts
	932	IGCM_debug_Print 1 ${host[$i]}
	933	i=$((i+1))
	934	done
	935	rm -f hosts.tmp
	936
	937	listnodes=${host[*]}
	938
	939	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
	940
	941	# Initialisation
[1300]	942	rank=0
	943	current_core=0
	944	core_per_node=4
[664]	945	init_exec=n
	946
[1300]	947	# Loop on the components
[664]	948	for comp in ${config_ListOfComponents[*]} ; do
	949
[1300]	950	eval ExeNameIn=\${config_Executable_${comp}[0]}
	951	eval ExeNameOut=\${config_Executable_${comp}[1]}
[664]	952
[1300]	953	# Not possible if oasis has an executable (i.e old version of oasis3)
	954	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	955	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	956	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	957	IGCM_debug_Verif_Exit
[664]	958	fi
	959
	960	# Only if we really have an executable for the component :
[1300]	961	if [ "X${ExeNameOut}" != X\"\" ] ; then
[664]	962
	963	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	964	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	965
	966	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	967	echo "" >> script_${ExeNameOut}.ksh
[1300]	968	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	969
	970	# Check if the number of threads is correct
	971	case ${comp_proc_omp_loc} in
	972	2\|4)
	973	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	974	;;
	975	*)
	976	IGCM_debug_Exit "ERROR with OMP parameters !"
	977	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	978	IGCM_debug_Print 2 "Only 2,4 as number of OMP threads are possible "
	979	IGCM_debug_Verif_Exit
	980	;;
	981	esac
	982	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	983	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	984	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	985	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	986	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	987	fi
[1318]	988
[1317]	989	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1318]	990	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	991	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	992	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
[664]	993	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	994
[1300]	995	if [ ${init_exec} = y ] ; then
	996	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	997	else
	998	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	999	init_exec=y
	1000	fi
[664]	1001
[1300]	1002	# Build rankfile : method used to assign cores and nodes for the MPI process
	1003	# Ex :
	1004	#rank 0=curie5296 slot=0,1,2,3
	1005	#rank 1=curie5296 slot=4,5,6,7
	1006	# Example of final command :
	1007	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1008	# with script_lmdz.x.ksh :
	1009	# #!/bin/ksh
	1010	#export OMP_STACKSIZE=200M
	1011	#export OMP_NUM_THREADS=4
	1012	#./lmdz.x
[664]	1013
[1300]	1014	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	1015	(( index_host = current_core / core_per_node ))
	1016	host_value=${host[${index_host}]}
	1017	(( slot = current_core % core_per_node ))
	1018	virg=","
	1019	string_final=""
	1020	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1021	string=$index$virg
	1022	string_final=$string_final$string
	1023	done
	1024	string_final=$( echo $string_final \| sed "s/.$//" )
	1025	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1026	(( rank = rank + 1 ))
	1027	(( current_core = current_core + comp_proc_omp_loc ))
[664]	1028	done
	1029	fi
	1030	done
[2]	1031	fi
	1032
[1300]	1033	# Only one executable (SPMD mode).
	1034	else
[664]	1035
	1036	for comp in ${config_ListOfComponents[*]} ; do
	1037
	1038	# Only if we really have an executable for the component :
	1039	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1040	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
	1041
	1042	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1043	echo "" >> script_${ExeNameOut}.ksh
	1044	if ( ${OK_PARA_OMP} ) ; then
	1045	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1300]	1046	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1047	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[664]	1048	fi
[1484]	1049
	1050	if [ ${mpiTasks} -eq 1 ] && [ ${openMPthreads} -eq 1 ] ; then
	1051	# Case with only one proc
	1052	# mpirun should not be used at obelix for only one proc
	1053	# Output text per process
	1054	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1055	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1056	EXECUTION="time ./script_${ExeNameOut}.ksh"
	1057	elif ( ${OK_PARA_MPI} ) ; then
[664]	1058	# Default : mpirun used if nb_proc gt 1
[1484]	1059	# Output text per process
[1317]	1060	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
[664]	1061	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1062	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
[1484]	1063	elif [ ${executionType} -eq 6 ] ; then
	1064	# Sequentiel case: this is the same as the first case.
[1317]	1065	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
[664]	1066	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1067	EXECUTION="time ./script_${ExeNameOut}.ksh"
[1484]	1068	else
	1069	# Strange case.
	1070	IGCM_debug_Exit "ERROR with OK_PARA_MPI"
	1071	IGCM_debug_Print 2 "sys Obelix : OK_PARA_MPI=False and mpiTask/openMPthreds different from 1"
	1072	IGCM_debug_Verif_Exit
[664]	1073	fi
[1178]	1074
[1484]	1075
[1178]	1076	IGCM_debug_Print 1 "sys $( hostname ) : script_${ExeNameOut}.ksh contains"
	1077	cat script_${ExeNameOut}.ksh
	1078
[664]	1079	fi
[1180]	1080
[664]	1081	done
	1082
	1083	fi
	1084
[811]	1085	IGCM_debug_Print 1 "sys $( hostname ) : execution command is"
	1086	IGCM_debug_Print 1 "$EXECUTION"
[664]	1087
	1088	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[2]	1089	}
	1090
[933]	1091	#D-#==================================================
	1092	#D-function IGCM_sys_check_path
	1093	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1094	#D-* do not point to an important use directory. Stop immediately in that case.
	1095	#D-* Examples:
	1096	#D-
	1097	function IGCM_sys_check_path {
	1098	IGCM_debug_PushStack "IGCM_sys_check_path"
	1099	if ( $DEBUG_sys ) ; then
	1100	echo "IGCM_sys_check_path"
	1101	fi
[811]	1102
[933]	1103	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
	1104	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1105	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1106	IGCM_debug_Exit "This will stop the job"
[933]	1107	fi
	1108	IGCM_debug_PopStack "IGCM_sys_check_path"
	1109	}
	1110
[811]	1111	#D-#==================================================
	1112	#D-function IGCM_sys_check_quota. Dummy call here
	1113	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1114	#D-* Examples:
	1115	#D-
[664]	1116	function IGCM_sys_check_quota {
	1117	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1118	if ( $DEBUG_sys ) ; then
	1119	echo "IGCM_sys_check_quota"
	1120	fi
[1300]	1121
	1122	# TO BE IMPLEMENTED
	1123
[664]	1124	IGCM_debug_PopStack "IGCM_sys_check_quota"
	1125	}
	1126
[811]	1127	#D-#==================================================
[1286]	1128	#D-function IGCM_sys_projectAccounting
	1129	#D-* Purpose: store project accounting information in a file
	1130	#D-* Examples:
	1131	#D-
	1132	function IGCM_sys_projectAccounting {
	1133	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1134	if ( $DEBUG_sys ) ; then
	1135	echo "IGCM_sys_check_quota"
	1136	fi
	1137
	1138	touch $1
	1139
	1140	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1141	}
	1142
	1143	#D-#==================================================
[1244]	1144	#D-function IGCM_sys_getJobSchedulerID
	1145	#D-* Purpose: Get the job ID during execution
	1146	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1147	#D-
	1148	function IGCM_sys_getJobSchedulerID {
	1149	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1150	if ( $DEBUG_sys ) ; then
	1151	echo "IGCM_sys_getJobSchedulerID"
	1152	fi
	1153
[1300]	1154	# N/A content
[1244]	1155	eval ${1}="N/A"
	1156
	1157	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1158	}
	1159
	1160	#D-#==================================================
[1300]	1161	#D-function IGCM_sys_GetJobID
	1162	#D-* Purpose: Get the job ID from the JobName
	1163	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1164	#D-
	1165	function IGCM_sys_GetJobID {
	1166	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1167	if ( $DEBUG_sys ) ; then
	1168	echo "IGCM_sys_GetJobID"
	1169	fi
	1170
	1171	# N/A content
	1172	eval ${3}="N/A"
	1173	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1174	}
	1175
	1176	#D-#==================================================
[811]	1177	#D-function IGCM_sys_CountJobInQueue
[1244]	1178	#D-* Purpose: Count number of users job
[811]	1179	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1180	#D-
	1181	function IGCM_sys_CountJobInQueue {
	1182	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1183	if ( $DEBUG_sys ) ; then
	1184	echo "IGCM_sys_CountJobInQueue"
	1185	fi
	1186	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1187	}
[1272]	1188
	1189	#D-#==================================================
	1190	#D-function IGCM_sys_atlas
	1191	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1192	#D-* Examples:
	1193	#D-
	1194	function IGCM_sys_atlas {
	1195	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1196	if ( $DEBUG_sys ) ; then
	1197	echo "IGCM_sys_atlas :" $@
	1198	fi
	1199
	1200	typeset status
	1201
	1202	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1203	status=$?
	1204	if [ ${status} -gt 0 ] ; then
	1205	echo "IGCM_sys_atlas : error code ${status}"
	1206	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1207	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1208	IGCM_debug_PopStack "IGCM_sys_atlas"
	1209	return 1
	1210	else
	1211	IGCM_debug_PopStack "IGCM_sys_atlas"
	1212	return 0
	1213	fi
	1214
	1215	IGCM_debug_PopStack "IGCM_sys_atlas"
	1216	}
	1217
[1380]	1218	#D-#==================================================
	1219	#D-function IGCM_sys_rebuild_nemo
	1220	#D-* Purpose: rebuild nemo parallel files with general rebuild
	1221	#D-* Examples:
	1222	#D-
	1223
	1224	function IGCM_sys_rebuild_nemo {
	1225	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1226	if ( $DEBUG_sys ) ; then
	1227	echo "IGCM_sys_rebuild_nemo :" $@
	1228	fi
	1229
	1230	nemo_generic_restart_file_name_out=${1}
	1231	nemo_extension_out=${3}
	1232	shift ; shift ; shift
	1233
	1234	IGCM_sys_rebuild -o ${nemo_generic_restart_file_name_out}.${nemo_extension_out} $@
	1235
	1236	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1237	}
	1238

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_default.ksh @ 1624

Download in other formats: