source: branches/libIGCM_CESMEP/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1586

Last change on this file since 1586 was 1585, checked in by ssenesi, 13 months ago

merge branch libIGCM_CESMEP with trunk that handles RedHat8 on Irene
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 60.0 KB
Line 
1#!/bin/ksh
2
3#**************************************************************
4# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6# $Revision::                                          $ Revision of last commit
7# $Author::                                            $ Author of last commit
8# $Date::                                              $ Date of last commit
9# IPSL (2006)
10#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11#
12#**************************************************************
13
14#=========================================================
15# The documentation of this file can be automatically generated
16# if you use the prefix #D- for comments to be extracted.
17# Extract with command: cat lib* | grep "^#D-" | cut -c "4-"
18#=========================================================
19
20#D-#==================================================
21#D-LibIGCM_sys for Jean-Zay
22#D-#==================================================
23#D-
24#D- This ksh library if a layer under some usefull
25#D-environment variables and shell commands.
26#D-All those definitions depend on host particularities.
27#D-It manages a stack mechanism and test validity of operations.
28#D-All function described bellow must be prefixed by IGCM_sys.
29
30#====================================================
31# libIGCM_sys PARAMETERS
32#====================================================
33
34#====================================================
35# set DEBUG_sys to true to output calls of function
36typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38#====================================================
39# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
41  typeset -r DRYRUN=0
42  typeset -r DRYRUN_DEBUG=4
43else
44  typeset -r DRYRUN=${DRYRUN:=0}
45  typeset -r DRYRUN_DEBUG=0
46fi
47
48# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
49# -------------------------------------------------------------------------------------
50# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
51# |          |  Cp/Exe/param/files |            |         |                           |
52# |          |  Chmod Qsub         |            |         |                           |
53# -------------------------------------------------------------------------------------
54# |    0     |       yes           |    yes     |  yes    |      yes                  |
55# -------------------------------------------------------------------------------------
56# |    1     |       yes           |    yes     |  yes    |      no                   |
57# -------------------------------------------------------------------------------------
58# |    2     |       yes           |    yes     |  no     |      no                   |
59# -------------------------------------------------------------------------------------
60# |    3     |       yes           |    no      |  no     |      no                   |
61# -------------------------------------------------------------------------------------
62# |    4     |       yes           |    yes     |  creation of RUN_DIR and Job_debug  |
63# -------------------------------------------------------------------------------------
64
65
66#=====================================================
67# Global Variables :
68#=====================================================
69# Language : "fr" or "en"
70typeset -r MYLANG="fr"
71
72#=====================================================
73# Host user names project maxCpuTime
74# $hostname ou hostname
75typeset HOST=${HOST:=$( hostname )}
76# $username ou whoami
77typeset LOGIN=${LOGIN:=$( whoami )}
78# $hostname of the MASTER job
79typeset MASTER=jeanzay
80# add default project set by ins_job
81typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} | cut -c "1-3" ) 
82### jobWarningDelay in seconds
83##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
84 
85#D-
86#D-#==================================================
87#D-Program used in libIGCM
88#D-#==================================================
89
90# Submit command
91typeset SUBMIT=${SUBMIT:=sbatch}
92# rsync with path
93typeset -r RSYNC=/usr/bin/rsync
94# RSYNC_opt args to rsync
95typeset -r RSYNC_opt="-va"
96# ie storage filesystem
97typeset -r STOREHOST=${MASTER}
98typeset -r REMOTE_RSYNC=/usr/bin/rsync
99
100#====================================================
101# Access to module command
102#====================================================
103# See ticket #348
104# Following path should be used according to assist IDRIS
105. /gpfslocalsup/spack_soft/environment-modules/current/init/ksh
106# Following old path gives error messages while loading modules but it still works
107#. /usr/share/Modules/init/ksh
108
109#====================================================
110# Set environment tools (ferret, nco, cdo, rebuild, ...)
111#====================================================
112if [ X${TaskType} = Xcomputing ] ; then
113  IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
114else
115  module purge > /dev/null 2>&1
116  . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
117
118  . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
119
120#  export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
121  export UVCDAT_ANONYMOUS_LOG=FALSE
122fi
123
124# FYI
125[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
126[ ! X${TaskType} = Xchecking ] && module list
127
128# For AddNoise (not yet AddPertu3DOCE)
129export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/AddNoise/bin
130
131# For rebuild
132export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
133
134#====================================================
135# Host specific DIRECTORIES
136#====================================================
137
138#====================================================
139#- MirrorlibIGCM for frontend
140typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
141
142#====================================================
143#- libIGCM_POST for frontend
144typeset -r libIGCM_POST=${libIGCM}
145
146#====================================================
147#- R_EXE   (==> BIN_DIR = ${MODIPSL}/bin )
148typeset -r R_EXE="${MODIPSL}/bin"
149
150#====================================================
151#- SUBMIT_DIR : submission dir
152typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
153
154#====================================================
155#- IN
156typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
157
158#====================================================
159#- RUN_DIR_PATH : Temporary working directory (=> TMP)
160typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
161
162#====================================================
163#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
164typeset -r OUTCOMMAND_PATH=/tmp
165
166#====================================================
167#- HOST_MPIRUN_COMMAND
168typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
169
170#====================================================
171#- Max number of arguments passed to nco operator or demigration command
172UNIX_MAX_LIMIT=360
173
174#====================================================
175#- set PackDefault to true
176PackDefault=true
177
178#====================================================
179#- Default number of MPI task for IPSL coupled model
180#- required for backward compatibility
181#-
182DEFAULT_NUM_PROC_OCE=5
183DEFAULT_NUM_PROC_CPL=1
184(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
185
186#====================================================
187#- Number of computing cores per node
188#-
189NB_CORE_PER_NODE=40
190
191#D-#==================================================
192#D-function IGCM_sys_defineArchives
193#D-* Purpose:
194#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
195#D-* Define ARCHIVE : Dedicated to large files
196#D-* Define STORAGE : Dedicated to small/medium files
197#D-* Define R_OUT   : Output tree located on ARCHIVE
198#D-* Define R_FIG   : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
199#D-* Define R_BUF   : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
200#D-* if SpaceName=TEST everything is stored on SCRATCH
201#D-* Examples:
202#D-
203function IGCM_sys_defineArchives {
204  IGCM_debug_PushStack "IGCM_sys_defineArchives"
205
206  # Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
207  if [ X${config_UserChoices_DataProject} = X ] || [ X${config_UserChoices_DataProject} = DEFAULT ]; then
208    # The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
209    # Default option: Set DataProject according to the project used for submission
210    DataProject=${PROJECT}
211  else
212    # Use the project set by the variable DataProject in section UserChoices in config.card
213    IGCM_debug_Print 1 "Use DataProject as set in config.card : ${config_UserChoices_DataProject}"
214    DataProject=${config_UserChoices_DataProject}
215  fi
216  IGCM_debug_Print 1 "Data project for output to be used: ${DataProject}"
217
218  #====================================================
219  #- RUN_DIR_PATH : Temporary working directory (=> TMP)
220  #- DataProject = for is the dataproject for Training account at Idris
221  #- they are not store in rech but in another partition
222  if [ X${DataProject} = Xfor  ]; then
223      typeset -r RUN_DIR_PATH=$SCRATCH/RUN_DIR/${SLURM_JOBID}_${$}
224  else
225      typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/gpfsscratch/rech/${DataProject}/${LOGIN}/RUN_DIR/${SLURM_JOBID}_${$}}
226  fi
227
228  if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
229    #====================================================
230    #- ARCHIVE (dedicated to large files)
231    ARCHIVE=${config_UserChoices_ARCHIVE}
232  else
233      if [ X${DataProject} = Xfor  ]; then
234          ARCHIVE=$STORE
235      else
236          #====================================================
237          #- ARCHIVE (dedicated to large files)
238          ARCHIVE=/gpfsstore/rech/${DataProject}/${LOGIN}
239      fi
240  fi
241
242  if [ ! X${config_UserChoices_STORAGE} = X ]; then
243    #====================================================
244    #- STORAGE (dedicated to small/medium files)
245    STORAGE=${config_UserChoices_STORAGE}
246  else
247    #====================================================
248    #- STORAGE (dedicated to small/medium files)
249      if [ X${DataProject} = Xfor  ]; then
250          STORAGE=$WORK
251      else
252          STORAGE=/gpfswork/rech/${DataProject}/${LOGIN}
253      fi
254  fi
255
256  if [ X${config_UserChoices_SpaceName} = XTEST ]; then
257    #====================================================
258    #- R_OUT
259    if [ X${DataProject} = Xfor  ]; then
260        R_OUT=$SCRATCH/IGCM_OUT
261    else
262        R_OUT=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
263    fi
264    #====================================================
265    #- R_FIG (hosting figures : monitoring and atlas, and/or small files)
266    R_FIG=${R_OUT}
267
268    IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
269
270  else
271    #====================================================
272    #- R_OUT
273    R_OUT=${ARCHIVE}/IGCM_OUT
274
275    #====================================================
276    #- R_FIG (hosting figures : monitoring and atlas, and/or small files)
277    R_FIG=${STORAGE}/IGCM_OUT
278  fi
279
280  #====================================================
281  #- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
282  CMIP6_BUF=${STORAGE}/IGCM_OUT
283
284  #====================================================
285  #- R_BUF (ONLY FOR double copy an scratch)
286 
287  if [ X${DataProject} = Xfor  ]; then
288      R_BUF=$SCRATCH/IGCM_OUT
289  else
290      R_BUF=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
291  fi
292
293  IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
294  IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
295  IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
296
297  IGCM_debug_PopStack "IGCM_sys_defineArchives"
298}
299
300#D-#==================================================
301#D-function IGCM_sys_RshArchive
302#D-* Purpose: Archive rsh command
303#D-* Examples:
304#D-
305function IGCM_sys_RshArchive {
306  IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
307  /bin/ksh <<-EOF
308    ${@}
309EOF
310  status=$?
311  if [ ${status} -gt 0 ] ; then
312    IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
313    IGCM_debug_Exit "IGCM_sys_RshArchive"
314  fi
315  IGCM_debug_PopStack "IGCM_sys_RshArchive"
316}
317
318#D-#==================================================
319#D-function IGCM_sys_RshArchive_NoError
320#D-* Purpose: Archive rsh command, without error
321#D-*          used only in monitoring.job
322#D-* Examples:
323#D-
324function IGCM_sys_RshArchive_NoError {
325  IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
326  /bin/ksh <<-EOF
327    ${@} 2> /dev/null
328EOF
329  IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
330}
331
332#D-#==================================================
333#D-function IGCM_sys_MkdirArchive
334#D-* Purpose: Mkdir on Archive
335#D-* Examples:
336#D-
337function IGCM_sys_MkdirArchive {
338  IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
339  if ( $DEBUG_sys ) ; then
340    echo "IGCM_sys_MkdirArchive :" $@
341  fi
342  #- creation de repertoire sur le serveur fichier
343  if [ ! -d ${1} ]; then
344    \mkdir -p $1
345    status=$?
346
347    if [ ${status} -gt 0 ] ; then
348      IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
349      IGCM_debug_Exit "IGCM_sys_MkdirArchive"
350    fi
351  fi
352  IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
353}
354
355#D-#==================================================
356#D-function IGCM_sys_TestDirArchive
357#D-* Purpose: Test Directory that must exists on Archive
358#D-* Examples:
359#D-
360function IGCM_sys_TestDirArchive {
361  IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
362  if ( $DEBUG_sys ) ; then
363    echo "IGCM_sys_TestDirArchive :" $@
364  fi
365  typeset ExistFlag
366  ExistFlag=$( [ -d $1 ] && echo 0 || echo 1 )
367  IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
368  return ${ExistFlag}
369}
370
371#D-#==================================================
372#D-function IGCM_sys_IsFileArchived
373#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
374#D-* Examples:
375#D-
376function IGCM_sys_IsFileArchived {
377  IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
378  if ( $DEBUG_sys ) ; then
379    echo "IGCM_sys_IsFileArchived :" $@
380  fi
381  typeset IsArchivedFlag
382# To be implemented on Jean Zay
383#  IsArchivedFlag=$( [ X$( echo $@ | grep ^\/ccc\/store ) != X ] && echo 0 || echo 1 )
384  IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
385
386  return ${IsArchivedFlag}
387}
388
389#D-#==================================================
390#D-function IGCM_sys_TestFileArchive
391#D-* Purpose: Test file that must NOT EXISTS on Archive
392#D-* Examples:
393#D-
394function IGCM_sys_TestFileArchive {
395  IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
396  typeset ExistFlag
397  ExistFlag=$( [ -f $1 ] && echo 0 || echo 1 )
398  IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
399
400  return ${ExistFlag}
401}
402
403#D-#==================================================
404#D-function IGCM_sys_CountFileArchive
405#D-* Purpose: Count files on Archive filesystem
406#D-* Examples:
407#D-
408function IGCM_sys_CountFileArchive {
409  IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
410  ls ${@} 2>/dev/null | wc -l
411  if [ $? -gt 0 ] ; then
412    echo "IGCM_sys_CountFileArchive : erreur."
413  fi
414  IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
415}
416
417#D-#==================================================
418#D-function IGCM_sys_Tree
419#D-* Purpose: Tree directories with files on ${ARCHIVE}
420#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
421#D-
422function IGCM_sys_Tree {
423  IGCM_debug_PushStack "IGCM_sys_Tree" $@
424  if ( $DEBUG_sys ) ; then
425    echo "IGCM_sys_Tree :" $@
426  fi
427
428  \ls -lR ${@}
429
430  IGCM_debug_PopStack "IGCM_sys_Tree"
431}
432
433#D-#==================================================
434#D-function IGCM_sys_Qsub
435#D-* Purpose: Qsub new job
436#D-* Examples:
437#D-
438function IGCM_sys_Qsub {
439  IGCM_debug_PushStack "IGCM_sys_Qsub" $@
440  if ( $DEBUG_sys ) ; then
441    echo "IGCM_sys_Qsub :" $@
442  fi
443  typeset options status
444  options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
445
446  /usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
447  status=$?
448
449  cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
450  if [ ${status} -gt 0 ] ; then
451    IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
452    IGCM_debug_Exit "IGCM_sys_Qsub"
453  else
454    JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
455  fi
456  IGCM_debug_PopStack "IGCM_sys_Qsub"
457}
458
459#D-#==================================================
460#D-function IGCM_sys_QsubPost
461#D-* Purpose: Qsub new job on scalaire
462#D-* Examples:
463#D-
464function IGCM_sys_QsubPost {
465  IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
466
467  BAK_DIR=$PWD 
468  cd ${SUBMIT_DIR}
469
470  if ( $DEBUG_sys ) ; then
471    echo "IGCM_sys_QsubPost :" $@
472  fi
473  typeset options status
474  options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
475
476  sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
477  status=$?
478
479  cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
480  if [ ${status} -gt 0 ] ; then
481    IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
482    IGCM_debug_Exit "IGCM_sys_QsubPost"
483  else
484    JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
485  fi
486
487  cd ${BAK_DIR}
488
489  IGCM_debug_PopStack "IGCM_sys_QsubPost"
490}
491
492#D-*************************
493#D- File transfer functions
494#D-*************************
495#D-
496
497#D-#==================================================
498#D-function IGCM_sys_RmRunDir
499#D-* Purpose: rm tmpdir (dummy function most of the time batch
500#D-                      scheduler will do the job)
501#D-* Examples:
502#D-
503function IGCM_sys_RmRunDir {
504  IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
505  if ( $DEBUG_sys ) ; then
506    echo "IGCM_sys_RmRunDir :" $@
507  fi
508
509  typeset status
510
511  echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
512  \rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
513  status=$?
514
515  if [ ${status} -gt 0 ] ; then
516    IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
517    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
518    IGCM_debug_Exit "IGCM_sys_RmRunDir"
519  else
520    \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
521  fi
522  IGCM_debug_PopStack "IGCM_sys_RmRunDir"
523}
524
525#D-#==================================================
526#D-function IGCM_sys_Put_Dir
527#D-* Purpose: Copy a complete directory on $(ARCHIVE)
528#D-* Examples:
529#D-
530function IGCM_sys_Put_Dir {
531  IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
532  if ( $DEBUG_sys ) ; then
533    echo "IGCM_sys_Put_Dir :" $@
534  fi
535  if [ $DRYRUN = 0 ]; then
536    if [ ! -d ${1} ] ; then
537      echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
538      IGCM_debug_PopStack "IGCM_sys_Put_Dir"
539      return
540    fi
541
542    typeset status
543
544    # Only if we use rsync
545    #IGCM_sys_TestDirArchive $( dirname $2 )
546    #
547    #USUAL WAY
548    \cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
549    status=$?
550
551    if [ ${status} -gt 0 ] ; then
552      IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
553      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
554      IGCM_debug_Exit "IGCM_sys_Put_Dir"
555    else
556      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
557    fi
558  fi
559  IGCM_debug_PopStack "IGCM_sys_Put_Dir"
560}
561
562#D-#==================================================
563#D-function IGCM_sys_Get_Dir
564#D-* Purpose: Copy a complete directory from ${ARCHIVE}
565#D-* Examples:
566#D-
567function IGCM_sys_Get_Dir {
568  IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
569  if ( $DEBUG_sys ) ; then
570    echo "IGCM_sys_Get_Dir :" $@
571  fi
572  if [ $DRYRUN = 0 ]; then
573    typeset NB_ESSAI DELAI status i
574    # number of tentative
575    NB_ESSAI=3
576    # time delay between tentative
577    DELAI=2
578
579    #
580    # USUAL WAY
581    # add here command to demigrate all offline files to reduce time of this command
582    # actually on Jean Zay files are not archive on bands
583
584    i=0
585    while [ $i -lt $NB_ESSAI ] ; do
586      \cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
587      status=$?
588      if [ ${status} -gt 0 ] ; then
589        IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
590        IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
591        sleep $DELAI
592      else
593        break
594      fi
595      (( i = i + 1 ))
596    done
597
598    if [ ${status} -gt 0 ] ; then
599      IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
600      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
601      IGCM_debug_Exit "IGCM_sys_Get_Dir"
602    else
603      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
604    fi
605  fi
606  IGCM_debug_PopStack "IGCM_sys_Get_Dir"
607}
608
609#D-#==================================================
610#D-function IGCM_sys_Put_Rest
611#D-* Purpose: Put computied restarts on ${ARCHIVE}.
612#D-           File and target directory must exist.
613#D-* Examples:
614#D-
615function IGCM_sys_Put_Rest {
616  IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
617  if ( $DEBUG_sys ) ; then
618    echo "IGCM_sys_Put_Rest :" $@
619  fi
620  if [ $DRYRUN = 0 ]; then
621    if [ ! -f ${1} ] ; then
622      echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
623      IGCM_debug_Exit "IGCM_sys_Put_Rest"
624    fi
625
626    typeset status
627    #
628    # USUAL WAY
629    \cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
630    status=$?
631
632#       #RSYNC WITH NETWORK SSH CALL
633#       echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
634#       ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
635
636#       #RSYNC WITH NFS USE
637#       echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
638#       ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
639
640#       status=$?
641#       IGCM_sys_Rsync_out $status
642
643#       ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
644#       (( status=status+$? ))
645
646    if [ ${status} -gt 0 ] ; then
647      IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
648      [ -f ${1} ] && ls -l ${1}
649      [ -f ${2} ] && ls -l ${2}
650      [ -f ${2}/${1} ] && ls -l ${2}/${1}
651      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
652      IGCM_debug_Exit "IGCM_sys_Put_Rest"
653    else
654
655      if [ X${JobType} = XRUN ] ; then
656        [ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
657        [ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
658      fi
659
660      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
661    fi
662  fi
663  IGCM_debug_PopStack "IGCM_sys_Put_Rest"
664}
665
666#D-#==================================================
667#D-function IGCM_sys_Put_Out
668#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
669#D-* Examples:
670#D-
671function IGCM_sys_Put_Out {
672  IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
673  if ( $DEBUG_sys ) ; then
674    echo "IGCM_sys_Put_Out :" $@
675  fi
676
677  typeset NB_ESSAI DELAI status i exist skip
678  typeset fileDeviceNumberInHex directoryDeviceNumberInHex
679
680  # number of tentative
681  NB_ESSAI=3
682  # time delay between tentative
683  DELAI=2
684
685  if [ $DRYRUN = 0 ]; then
686    if [ ! -f ${1} ] ; then
687      echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
688      IGCM_debug_PopStack "IGCM_sys_Put_Out"
689      return 1
690    fi
691    #
692    IGCM_sys_MkdirArchive $( dirname $2 )
693    #
694    exist=false
695    skip=false
696    if [ -f $2 ] ; then
697      IGCM_debug_Print 1 "$2 already exist"
698    # add here command to demigrate all offline files to reduce time of this command
699    # actually on Jean Zay files are not archive on bands
700
701      exist=true
702      if [ "X$( diff $1 $2 )" = X ] ; then
703        IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
704        skip=true
705      else
706        IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
707        skip=false
708      fi
709    fi
710    #
711    if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
712      IGCM_sys_Chmod u+w $2
713    fi
714
715    if [ X${skip} = Xfalse ] ; then
716      i=0
717      while [ $i -lt $NB_ESSAI ] ; do
718        # Identify file system
719        fileDeviceNumberInHex=$( stat -c %d $1 )
720        status=$?
721        if [ ${status} -gt 0 ] ; then
722          IGCM_debug_Exit "IGCM_sys_Put_Out"
723        fi
724        # Identify file system
725        directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
726        status=$?
727        if [ ${status} -gt 0 ] ; then
728          IGCM_debug_Exit "IGCM_sys_Put_Out"
729        fi
730
731        if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] || [ X$3 = XNOMOVE ] ) ; then
732          # They are not on the same device. USUAL WAY
733          \cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
734          status=$?
735        else
736          # They are on the same device. NOT SO USUAL WAY
737          \mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
738          status=$?
739        fi
740        if [ ${status} -gt 0 ]; then
741          IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
742          IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
743          [ -f ${1} ] && ls -l ${1}
744          [ -f ${2} ] && ls -l ${2}
745          [ -f ${2}/${1} ] && ls -l ${2}/${1}
746          sleep $DELAI
747        else
748          break
749        fi
750        (( i = i + 1 ))
751      done
752    fi
753
754#       #RSYNC WITH NETWORK SSH CALL
755#       echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
756#       ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
757
758#       #RSYNC WITH NFS USE
759#       echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
760#       ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
761
762#       status=$?
763#       IGCM_sys_Rsync_out $status
764
765#       ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
766#       (( status=status+$? ))
767
768    if [ ${status} -gt 0 ] ; then
769      IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
770      [ -f ${1} ] && ls -l ${1}
771      [ -f ${2} ] && ls -l ${2}
772      [ -f ${2}/${1} ] && ls -l ${2}/${1}
773      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
774      IGCM_debug_Exit "IGCM_sys_Put_Out"
775    else
776
777      if [ X${JobType} = XRUN ] ; then
778        if [ X${3} = X ] ; then
779          [ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
780          [ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
781        fi
782      fi
783
784      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
785    fi
786  fi
787  IGCM_debug_PopStack "IGCM_sys_Put_Out"
788  return 0
789}
790
791#D-#==================================================
792#D-function IGCM_sys_Get
793#D-* Purpose: Get a file from ${ARCHIVE}
794#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
795#D-            IGCM_sys_Get /l Array_contain_myfiles /destpath/
796function IGCM_sys_Get {
797  IGCM_debug_PushStack "IGCM_sys_Get" $@
798
799  typeset DEST dm_liste target file_work
800  typeset NB_ESSAI DELAI status i
801
802  if ( $DEBUG_sys ) ; then
803    echo "IGCM_sys_Get :" $@
804  fi
805
806  # number of tentative
807  NB_ESSAI=3
808  # time delay between tentative
809  DELAI=2
810
811  if [ $DRYRUN -le 2 ]; then
812    if [ X${1} = X'/l' ] ; then
813      eval set +A dm_liste \${${2}}
814    else
815      eval set +A dm_liste ${1}
816    fi
817    eval DEST=\${${#}}
818    # add here command to demigrate all offline files to reduce time of this command
819    # actually on Jean Zay files are not archive on bands
820
821    status=$?
822    if [ ${status} -gt 0 ] ; then
823      echo "WARNING IGCM_sys_Get : error code ${status}"
824      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
825      echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
826    fi
827
828    #if [ ${status} -gt 0 ] ; then
829    #  if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
830    #    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
831    #    echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
832    #    sleep 30
833    #    echo "We try another time"
834    ##    dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
835    #    ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
836    #    status=$?
837    #    if [ ${status} -gt 0 ] ; then
838    #      echo "ERROR IGCM_sys_Get : again demigration error :"
839    #      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
840    #      IGCM_debug_Exit "IGCM_sys_Get"
841    #    fi
842    #  else
843    #    echo "ERROR IGCM_sys_Get : demigration error :"
844    #    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
845    #    IGCM_debug_Exit "IGCM_sys_Get"
846    #  fi
847    #fi
848
849    #   #RSYNC WITH NETWORK SSH CALL
850    #   echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
851    #   ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
852
853    #   #RSYNC WITH NFS USE
854    #   echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
855    #   ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
856
857    #   status=$?
858    #   IGCM_sys_Rsync_out $status
859
860    #   ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
861    #   (( status=status+$? ))
862
863    #USUAL WAY
864    if [ X${1} = X'/l' ] ; then
865      for target in ${dm_liste[*]} ; do
866        local_file=$( basename ${target} )
867        # test if the target file is present before the loop
868        IGCM_sys_TestFileArchive ${target}
869        status=$?
870        if [ ${status} -gt 0 ] ; then
871          echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
872          IGCM_debug_Exit "IGCM_sys_Get"
873        else
874          i=0
875          while [ $i -lt $NB_ESSAI ] ; do
876            #if [ X${DoLink} = Xtrue ] ; then
877            #  \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
878            #  status=$?
879            #  else
880            #  \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
881            #  status=$?
882            #fi
883            \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
884            status=$?
885            if [ ${status} -gt 0 ]; then
886              IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
887              IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
888              sleep $DELAI
889            else
890              break
891            fi
892            (( i = i + 1 ))
893          done
894          if [ ${status} -gt 0 ] ; then
895            echo "IGCM_sys_Get : error"
896            cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
897            \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
898            IGCM_debug_Exit "IGCM_sys_Get"
899          else
900            \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
901          fi
902        fi
903      done
904    else
905      i=0
906      while [ $i -lt $NB_ESSAI ] ; do
907        \cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
908        status=$?
909        if [ ${status} -gt 0 ]; then
910          IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
911          IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
912          sleep $DELAI
913        else
914          break
915        fi
916        (( i = i + 1 ))
917      done
918      if [ ${status} -gt 0 ] ; then
919        echo "IGCM_sys_Get : error"
920        cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
921        \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
922        IGCM_debug_Exit "IGCM_sys_Get"
923      else
924        \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
925      fi
926    fi
927  fi
928  IGCM_debug_PopStack "IGCM_sys_Get"
929}
930
931#D-#==================================================
932#D-function IGCM_sys_GetDate_Monitoring
933#D-* Purpose: get the last year for which the monitoring has been computed
934#D-* Examples:
935#D-
936function IGCM_sys_GetDate_Monitoring {
937  IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
938  if ( $DEBUG_sys ) ; then
939    echo "IGCM_sys_GetDate_Monitoring :" $@
940  fi
941
942  eval ${2}=$( cdo showyear ${1} 2> /dev/null | gawk '{ print $NF }' )
943
944  IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
945}
946
947#D-#==================================================
948#D-function IGCM_sys_Dods_Rm
949#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
950#D-* Examples:
951#D-
952function IGCM_sys_Dods_Rm {
953  if ( $DEBUG_sys ) ; then
954    echo "IGCM_sys_Dods_Rm :" $@
955  fi
956  typeset status
957  if [ $DRYRUN = 0 ]; then
958
959#    if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
960#      echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
961#      echo "Nothing has been done."
962#      return
963#    fi
964
965    if [ "$#" -eq 1 ]; then
966      thredds_rm ${R_DODS}/${1}
967      status=$?
968    else
969        thredds_rm ${R_DODS}
970      status=$?
971    fi
972
973#    if [ ${status} -gt 0 ] ; then
974#      echo "IGCM_sys_Dods_Rm : error."
975#      cat out_dods_rm
976#      IGCM_debug_Exit "IGCM_sys_Dods_Rm"
977#    else
978#      rm out_dods_rm
979#    fi
980
981  fi
982  return $status
983}
984
985#D-#==================================================
986#D-function IGCM_sys_Dods_Cp
987#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
988#D-* Examples:
989#D-
990function IGCM_sys_Dods_Cp {
991  if ( $DEBUG_sys ) ; then
992    echo "IGCM_sys_Dods_Cp :" $@
993  fi
994  typeset status
995  if [ $DRYRUN = 0 ]; then
996
997#    if [ ! -d ${R_SAVE}/${1} ] ; then
998#      echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
999#      echo "Nothing has been done."
1000#      return
1001#    fi
1002
1003
1004      thredds_cp ${1} ./${R_DODS}/.
1005
1006    status=$?
1007
1008#       if [ ${status} -gt 0 ] ; then
1009#           echo "IGCM_sys_Dods_Cp : error."
1010#           cat out_dods_cp
1011#           IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1012#       else
1013#           rm out_dods_cp
1014#       fi
1015
1016  fi
1017  return $status
1018}
1019
1020#D-#==================================================
1021#D-function IGCM_sys_Put_Dods
1022#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1023#D-* Examples:
1024#D-
1025function IGCM_sys_Put_Dods {
1026  IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1027  if ( $DEBUG_sys ) ; then
1028    echo "IGCM_sys_Put_Dods :" $@
1029  fi
1030  typeset status
1031  if [ $DRYRUN = 0 ]; then
1032    if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1033      echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1034      echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1035      echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1036      IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1037      return
1038    fi
1039
1040    if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1041      echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1042      echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1043      echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1044      IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1045      return
1046    fi
1047    #
1048    if [ -d ${R_SAVE}/${1} ] ; then
1049      cd ${R_SAVE}
1050    elif [ -d ${R_FIGR}/${1} ] ; then
1051      cd ${R_FIGR}
1052    fi
1053
1054    IGCM_sys_Dods_Rm ${1}
1055    IGCM_sys_Dods_Cp ${1}
1056    status=0
1057
1058    if [ ${status} -gt 0 ] ; then
1059      echo "IGCM_sys_Put_Dods : error."
1060      IGCM_debug_Exit "IGCM_sys_Put_Dods"
1061    fi
1062  fi
1063  IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1064}
1065
1066##############################################################
1067# REBUILD OPERATOR
1068
1069#D-#==================================================
1070#D-function IGCM_sys_sync
1071#D-* Purpose: flush buffer on disk
1072#D-* Examples:
1073#D-
1074function IGCM_sys_sync {
1075  IGCM_debug_PushStack "IGCM_sys_sync" $@
1076  if ( $DEBUG_sys ) ; then
1077    echo "IGCM_sys_sync :" $@
1078  fi
1079
1080  /bin/sync
1081
1082  IGCM_debug_PopStack "IGCM_sys_sync"
1083}
1084
1085############################################################
1086# Activate Running Environnment Variables
1087
1088#D-#==================================================
1089#D-function IGCM_sys_activ_variables
1090#D-* Purpose: set environement variables prior to execution
1091#D-* Examples:
1092#D-
1093function IGCM_sys_activ_variables {
1094  IGCM_debug_PushStack "IGCM_sys_activ_variables"
1095  if ( $DEBUG_sys ) ; then
1096    echo "IGCM_sys_activ_variables"
1097  fi
1098
1099# --------------------------------------------------------------------
1100#D- MODULE specifications
1101# --------------------------------------------------------------------
1102  # Source the file EnvFile containing all module load needed to run the model.
1103  # EnvFile can be specified in config.card. If this is the case and if the file exists,
1104  # it will be used. Otherwise a default file will be used.
1105  if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1106      # EnvFile is set in config.card and the file exists
1107      IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1108      EnvFile=${config_UserChoices_EnvFile}
1109  else
1110      IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1111      module purge > /dev/null 2>&1
1112      EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1113  fi
1114  IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :" 
1115  IGCM_debug_Print 1 ${EnvFile}
1116  . ${EnvFile}
1117
1118  IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1119  module list
1120
1121# --------------------------------------------------------------------
1122#D- MPI specifications
1123# --------------------------------------------------------------------
1124
1125# --------------------------------------------------------------------
1126#D- Other specifications
1127# --------------------------------------------------------------------
1128
1129  ulimit -s unlimited
1130  if [ ${executionType} -eq 2 ] ; then
1131    export SLURM_HOSTFILE=./hostlist
1132  fi
1133
1134
1135  IGCM_debug_PopStack "IGCM_sys_activ_variables"
1136}
1137
1138############################################################
1139# Desactivate Running Environnment Variables
1140
1141#D-#==================================================
1142#D-function IGCM_sys_desactiv_variables
1143#D-* Purpose: unset environement variables after execution
1144#D-* Examples:
1145#D-
1146function IGCM_sys_desactiv_variables {
1147  IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1148  if ( $DEBUG_sys ) ; then
1149    echo "IGCM_sys_desactiv_variables"
1150  fi
1151# --------------------------------------------------------------------
1152#D- MPI specifications
1153# --------------------------------------------------------------------
1154  if [ ${executionType} -eq 2 ] ; then
1155    unset SLURM_HOSTFILE
1156  fi
1157# --------------------------------------------------------------------
1158#D- Other specifications
1159# --------------------------------------------------------------------
1160
1161  IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1162}
1163
1164############################################################
1165# Update job headers to be used by the scheduler
1166
1167#D-#==================================================
1168#D-function IGCM_sys_updateHeaders
1169#D-* Purpose: Update job headers to be used by the scheduler
1170#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1171#D-
1172function IGCM_sys_updateHeaders {
1173  IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1174  if ( $DEBUG_sys ) ; then
1175    echo "IGCM_sys_updateHeaders"
1176  fi
1177  typeset file
1178  file=$1
1179
1180  if [ ${executionType} -eq 1 ] ; then
1181    # MPMD + MPI
1182    sed -e "/::openMPthreads::/d"                  \
1183        -e "s/::JobNumProcTot::/${coreNumber}/"    \
1184      ${file} > ${file}.tmp
1185
1186  elif [ ${executionType} -eq 2 ] ; then
1187    # MPMD + MPI + OMP
1188      (( nodeNumber = coreNumber / NB_CORE_PER_NODE ))
1189      [ $(( ${coreNumber} % ${NB_CORE_PER_NODE} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
1190      sed -e "/::openMPthreads::/d"                 \
1191        -e "s/::JobNumProcTot::/${mpiTasks}/"       \
1192        -e "/ntasks/i\#SBATCH --nodes=${nodeNumber}"\
1193        -e "/ntasks/i\#SBATCH --exclusive"          \
1194         ${file} > ${file}.tmp
1195  elif [ ${executionType} -eq 3 ] ; then
1196    # SPMD + MPI/OMP
1197    sed -e "s/::openMPthreads::/${openMPthreads}/" \
1198        -e "s/::JobNumProcTot::/${mpiTasks}/"      \
1199      ${file} > ${file}.tmp
1200
1201  elif [ ${executionType} -eq 4 ] ; then
1202    # SPMD + MPI only
1203    sed -e "s/::JobNumProcTot::/${mpiTasks}/"      \
1204        -e "/::openMPthreads::/d"                  \
1205      ${file} > ${file}.tmp
1206
1207  elif [ ${executionType} -eq 5 ] ; then
1208    # SPMD + OMP only
1209    sed -e "s/::openMPthreads::/${openMPthreads}/" \
1210        -e "/::JobNumProcTot::/d"                  \
1211      ${file} > ${file}.tmp
1212
1213  elif [ ${executionType} -eq 6 ] ; then
1214    # SEQUENTIAL THEN
1215    sed -e "s/::JobNumProcTot::/1/"                \
1216        -e "/::openMPthreads::/d"                  \
1217      ${file} > ${file}.tmp
1218
1219  fi
1220
1221  IGCM_sys_Mv ${file}.tmp ${file}
1222
1223  IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1224}
1225
1226############################################################
1227# Build MPI/OMP scripts run file (dummy function)
1228
1229#D-#==================================================
1230#D-function IGCM_sys_build_run_file
1231#D-* Purpose: build run file (deprecated)
1232#D-* Examples:
1233#D-
1234function IGCM_sys_build_run_file {
1235  IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1236}
1237
1238############################################################
1239# Build MPI/OMP scripts
1240
1241#D-#==================================================
1242#D-function IGCM_sys_build_execution_scripts
1243#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1244#D-* Examples:
1245#D-
1246function IGCM_sys_build_execution_scripts
1247{
1248  IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1249  if ( $DEBUG_sys ) ; then
1250    echo "IGCM_sys_build_execution_scripts " $@
1251  fi
1252  IGCM_debug_Print 1 "executionType= ${executionType}"
1253  EXECUTION=${HOST_MPIRUN_COMMAND}
1254
1255  if ( ${OK_PARA_MPMD} ) ; then
1256
1257    # MPMD mode
1258    # 1 MPI only : executionType=1
1259    # 2 MPI/OpenMP : executionType=2
1260
1261    if [ -f run_file ] ; then
1262      IGCM_sys_Rm -f run_file
1263    fi
1264    touch run_file
1265
1266    # case 1 : Only MPI (MPMD)
1267    if  ( ! ${OK_PARA_OMP} ) ; then
1268      # Build run_file
1269      current_core=0
1270      # First loop on the components for the coupler ie oasis (only if oasis3)
1271      # the coupler ie oasis3 must be the first one
1272      for comp in ${config_ListOfComponents[*]} ; do
1273
1274        eval ExeNameIn=\${config_Executable_${comp}[0]}
1275        eval ExeNameOut=\${config_Executable_${comp}[1]}
1276
1277        # Only if the component has an executable
1278        if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1279
1280          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1281          (( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
1282          echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
1283          (( current_core = ${end_core} + 1 ))
1284        fi
1285      done
1286
1287      EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
1288
1289      IGCM_sys_Chmod u+x run_file
1290      if ( $DEBUG_sys ) ; then
1291        echo "run_file contains : "
1292        cat run_file
1293      fi
1294
1295    else 
1296
1297    # 2 MPI/OpenMP : executionType=2
1298
1299    # MPI-OpenMP (MPMD)
1300    # export SLURM_HOSTFILE=./hostlist
1301    # srun --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file
1302    # example of  hostlist file :
1303    # r3i3n33
1304    # r3i3n33
1305    # ...
1306    # example of run_file :
1307    # 0-70 ./prog_lmdz.x.sh %o %t
1308    # 71-430 ./prog_opa.xx.sh %o %t
1309    # 431-431 ./prog_xios.x.sh %o %t
1310    # examples of prog_file :
1311    # prog_lmdz.x.sh :
1312    # (( init = 0 + $1 ))
1313    # (( index = init * 10 ))
1314    # (( slot = index % 40 ))
1315    # taskset -c $slot-$((slot + 10 - 1)) ./script_lmdz.x.ksh
1316    # that will become
1317    # taskset -c 0-9 ./script_lmdz.x.ksh
1318    # ...
1319    # with script_lmdz.x.ksh
1320    # export OMP_STACKSIZE=3g
1321    # export OMP_PLACES=cores
1322    # export OMP_NUM_THREADS=10
1323    # ./lmdz.x > out_lmdz.x.out.${SLURM_PROCID} 2>out_lmdz.x.err.${SLURM_PROCID}
1324
1325    #  Hosts treatment
1326    _bkIFS=$IFS;
1327        IFS=$'\n'; set -f
1328        listnodes=($(< <( scontrol show hostnames $SLURM_JOB_NODELIST )))
1329        IFS=$_bkIFS; set +f
1330        rm -f hostlist
1331
1332     # Loop on the components to build run_file and script_exec files
1333        rank=0
1334        current_core=0
1335        current_core_mpi=0
1336
1337      for comp in ${config_ListOfComponents[*]} ; do
1338
1339        eval ExeNameIn=\${config_Executable_${comp}[0]}
1340        eval ExeNameOut=\${config_Executable_${comp}[1]}
1341
1342        # Not possible if oasis has an executable (i.e old version of oasis3)
1343        if ( [ "X${ExeNameOut}" != X\"\" ] &&  [ "X${comp}" = "XCPL" ] ) ; then
1344          IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1345          IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1346          IGCM_debug_Verif_Exit
1347        fi
1348
1349        # Only if we really have an executable for the component :
1350        if [ "X${ExeNameOut}" != X\"\" ] ; then
1351
1352          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1353          eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1354          eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1355     
1356
1357          # Build script files
1358
1359          echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1360          echo ""  >> script_${ExeNameOut}.ksh
1361          if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1362          # Check if the number of threads is correct
1363              case ${comp_proc_omp_loc} in
1364                  2|4|5|10|20)
1365                      IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1366                      ;;
1367                  *)
1368                      IGCM_debug_Exit "ERROR with OMP parameters !"
1369                      IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1370                      IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1371                      IGCM_debug_Verif_Exit
1372                      ;;
1373              esac
1374            echo "export OMP_STACKSIZE=3g"  >> script_${ExeNameOut}.ksh
1375            echo "export OMP_PLACES=cores"  >> script_${ExeNameOut}.ksh
1376            echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}"  >> script_${ExeNameOut}.ksh
1377
1378          fi
1379
1380          # to have out/err per process on different files
1381          echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1382
1383          IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1384
1385        # Build run_file
1386        # Only if the component has an executable
1387        if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1388
1389          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1390          (( end_core = ${current_core_mpi} + ${comp_proc_mpi_loc} - 1 ))
1391          echo "${current_core_mpi}-${end_core} ./prog_${ExeNameOut}.sh %o %t" >> run_file
1392          (( current_core_mpi = ${end_core} + 1 ))
1393        fi
1394
1395          if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1396              (( offset_comp_proc_loc =  NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1397          else
1398              (( offset_comp_proc_loc =  comp_proc_omp_loc ))
1399          fi
1400
1401          # Build configuration file
1402 
1403              echo "#!/bin/sh" > prog_${ExeNameOut}.sh
1404          echo "(( init = $current_core + \$1 ))" >> prog_${ExeNameOut}.sh
1405          echo "(( index = init * $comp_proc_omp_loc ))" >> prog_${ExeNameOut}.sh
1406          echo "(( slot = index % 40 ))" >> prog_${ExeNameOut}.sh
1407              echo "echo ${ExeNameOut} taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1))" >> prog_${ExeNameOut}.sh
1408          echo "taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1)) ./script_${ExeNameOut}.ksh" >> prog_${ExeNameOut}.sh
1409
1410          IGCM_sys_Chmod u+x prog_${ExeNameOut}.sh
1411
1412        # Build hostlist file
1413
1414          for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1415             (( index_host = current_core / NB_CORE_PER_NODE ))
1416             host_value=${listnodes[${index_host}]}
1417             echo "$host_value" >> hostlist
1418             if [ ${DRYRUN_DEBUG} = 4 ] ; then
1419                 echo "node_${index_host}_X" >> hostlist_template
1420             fi
1421             (( current_core = current_core + offset_comp_proc_loc ))
1422          done
1423        fi
1424      done
1425
1426      ## variable added to stop after 60s instead of 600s by default.
1427      ## This is used when no error comes from executables and when something stopped an executable without notice.
1428      export SLURM_WAIT=60
1429
1430      EXECUTION="${HOST_MPIRUN_COMMAND} --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file"
1431
1432      IGCM_sys_Chmod u+x run_file
1433      if ( $DEBUG_sys ) ; then
1434        echo "run_file contains : "
1435        cat run_file
1436      fi
1437
1438    fi # if ${OK_PARA_MPMD}
1439
1440  else
1441  # Only one executable (SPMD mode):  executionType=3, 4, 5 and 6
1442
1443    for comp in ${config_ListOfComponents[*]} ; do
1444
1445      # Only if we really have an executable for the component :
1446      eval ExeNameOut=\${config_Executable_${comp}[1]}
1447      if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1448
1449        # Build script files
1450
1451        echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1452        echo ""  >> script_${ExeNameOut}.ksh
1453        IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1454
1455        if ( ${OK_PARA_OMP} ) ; then
1456            eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1457            # Check if the number of threads is correct
1458            case ${comp_proc_omp_loc} in
1459                2|4|5|10|20)
1460                    IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1461                    ;;
1462                *)
1463                    IGCM_debug_Exit "ERROR with OMP parameters !"
1464                    IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1465                    IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1466                    IGCM_debug_Verif_Exit
1467                    ;;
1468            esac
1469            echo ""  >> script_${ExeNameOut}.ksh
1470            echo "export OMP_STACKSIZE=3g"  >> script_${ExeNameOut}.ksh
1471            echo "export OMP_PLACES=cores"  >> script_${ExeNameOut}.ksh
1472            echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1473        fi
1474
1475        eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1476       
1477        # To have out/err per process on different files
1478        echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}"  >> script_${ExeNameOut}.ksh
1479        EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1480
1481        IGCM_debug_Print 1 "sys Jean-Zay : script_${ExeNameOut}.ksh contains"
1482        cat script_${ExeNameOut}.ksh
1483
1484      fi
1485
1486    done
1487
1488  fi # ${OK_PARA_MPMD}
1489
1490  IGCM_debug_Print 1 "sys Jean-Zay : execution command is "
1491  IGCM_debug_Print 1 "$EXECUTION"
1492
1493  IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1494}
1495
1496#D-#==================================================
1497#D-function IGCM_sys_check_path
1498#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1499#D-* do not point to an important use directory. Stop immediately in that case.
1500#D-* Examples:
1501#D-
1502function IGCM_sys_check_path {
1503  IGCM_debug_PushStack "IGCM_sys_check_path"
1504  if ( $DEBUG_sys ) ; then
1505    echo "IGCM_sys_check_path"
1506  fi
1507
1508  if ( [ X${RUN_DIR_PATH} = X${HOME} ] || [ X${RUN_DIR_PATH} = X${WORK} ] || [ X${RUN_DIR_PATH} = X${SCRATCH} ] || [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1509    IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1510    IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1511    IGCM_debug_Exit "This will stop the job"
1512  elif ( [ X${RUN_DIR_PATH} = X${HOME} ] || [ X${RUN_DIR_PATH} = X/gpfswork/rech/${DataProject}/${LOGIN} ] || [ X${RUN_DIR_PATH} = X/gpfsscratch/rech/${DataProject}/${LOGIN} ] || [ X${RUN_DIR_PATH} = X/gpfsstore/rech/${DataProject}/${LOGIN} ] ) ; then
1513    IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1514    IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1515    IGCM_debug_Exit "This will stop the job"
1516  fi
1517  IGCM_debug_PopStack "IGCM_sys_check_path"
1518}
1519
1520#D-#==================================================
1521#D-function IGCM_sys_check_quota
1522#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1523#D-* Examples:
1524#D-
1525function IGCM_sys_check_quota {
1526  IGCM_debug_PushStack "IGCM_sys_check_quota"
1527  echo "IGCM_sys_check_quota is desactivated on JeanZay because there is actually no way to check scratch quota on this computing center (2019.11.19)"
1528#  if ( $DEBUG_sys ) ; then
1529#    echo "IGCM_sys_check_quota"
1530#  fi
1531#  # Limit of quota (in %)
1532#  limit_quota=90
1533#
1534#  # Check of the volume
1535#  volume_quota=$(ccc_quota | grep ' scratch' | gawk '{print $2}')
1536#  volume_avail=$(ccc_quota | grep ' scratch' | gawk '{print $3}')
1537#
1538#  if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1539#
1540#    unit_avail=${volume_avail: -1}
1541#    unit_quota=${volume_quota: -1}
1542#
1543#    if [ "${unit_quota}" = "*" ] ; then
1544#      IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1545#      IGCM_debug_Print 1 "More than 100% of your quota is used"
1546#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1547#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1548#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1549#      IGCM_debug_Verif_Exit
1550#    fi
1551#
1552#    temp_avail=${volume_avail%%${volume_avail: -1}*}
1553#    temp_quota=${volume_quota%%${volume_quota: -1}*}
1554#
1555#    if [ ! ${unit_avail} = ${unit_quota} ] ; then
1556#
1557#    # Convertion
1558#      if [ ${volume_avail: -1} = "T" ] ; then
1559#        (( temp_avail = temp_avail * 1000000000000 ))
1560#      elif [ ${volume_avail: -1} = "G" ] ; then
1561#        (( temp_avail = temp_avail * 1000000000 ))
1562#      elif [ ${volume_avail: -1} = "M" ] ; then
1563#        (( temp_avail = temp_avail * 1000000 ))
1564#      elif [ ${volume_avail: -1} = "k" ] ; then
1565#        (( temp_avail = temp_avail * 1000 ))
1566#      else
1567#        (( temp_avail = volume_avail ))
1568#      fi
1569#      if [ ${volume_quota: -1} = "T" ] ; then
1570#        (( temp_quota = temp_quota * 1000000000000 ))
1571#      elif [ ${volume_quota: -1} = "G" ] ; then
1572#        (( temp_quota = temp_quota * 1000000000 ))
1573#      elif [ ${volume_quota: -1} = "M" ] ; then
1574#        (( temp_quota = temp_quota * 1000000 ))
1575#      elif [ ${volume_quota: -1} = "k" ] ; then
1576#        (( temp_quota = temp_quota * 1000 ))
1577#      else
1578#        (( temp_quota = volume_quota ))
1579#      fi
1580#    fi
1581#
1582#    quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" | bc)
1583##    echo "volume ratio is " $quota_volume
1584#
1585#    if [ ${quota_volume} -ge ${limit_quota} ] ; then
1586#      IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1587#      IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1588#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1589#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1590#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1591#      IGCM_debug_Verif_Exit
1592#    fi
1593#
1594#  fi
1595#
1596## Check of the number of inodes
1597#
1598#  inode_quota=$(ccc_quota | grep ' scratch' | gawk '{print $6}')
1599#  inode_avail=$(ccc_quota | grep ' scratch' | gawk '{print $7}')
1600#
1601#  if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1602#
1603#    unit_avail=${inode_avail: -1}
1604#    unit_quota=${inode_quota: -1}
1605#
1606#    if [ "${unit_quota}" = "*" ] ; then
1607#      IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1608#      IGCM_debug_Print 1 "More than 100% of your quota is used"
1609#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1610#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1611#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1612#      IGCM_debug_Verif_Exit
1613#    fi
1614#
1615#    temp_avail=${inode_avail%%${inode_avail: -1}*}
1616#    temp_quota=${inode_quota%%${inode_quota: -1}*}
1617#
1618#    if [ ! ${unit_avail} = ${unit_quota} ] ; then
1619#
1620#    # Convertion
1621#      if [ ${inode_avail: -1} = "T" ] ; then
1622#        (( temp_avail = temp_avail * 1000000000000 ))
1623#      elif [ ${inode_avail: -1} = "G" ] ; then
1624#        (( temp_avail = temp_avail * 1000000000 ))
1625#      elif [ ${inode_avail: -1} = "M" ] ; then
1626#        (( temp_avail = temp_avail * 1000000 ))
1627#      elif [ ${inode_avail: -1} = "k" ] ; then
1628#        (( temp_avail = temp_avail * 1000 ))
1629#      else
1630#        (( temp_avail = inode_avail ))
1631#      fi
1632#
1633#      if [ ${inode_quota: -1} = "T" ] ; then
1634#        (( temp_quota = temp_quota * 1000000000000 ))
1635#      elif [ ${inode_quota: -1} = "G" ] ; then
1636#        (( temp_quota = temp_quota * 1000000000 ))
1637#      elif [ ${inode_quota: -1} = "M" ] ; then
1638#        (( temp_quota = temp_quota * 1000000 ))
1639#      elif [ ${inode_quota: -1} = "k" ] ; then
1640#        (( temp_quota = temp_quota * 1000 ))
1641#      else
1642#        (( temp_quota = inode_quota ))
1643#      fi
1644#    fi
1645#    quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" | bc)
1646##    echo "inode ratio is " $quota_inode
1647#
1648#    if [ ${quota_inode} -ge ${limit_quota} ] ; then
1649#      IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1650#      IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1651#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1652#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1653#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1654#      IGCM_debug_Verif_Exit
1655#    fi
1656#  fi
1657  IGCM_debug_PopStack "IGCM_sys_check_quota"
1658}
1659
1660#D-#==================================================
1661#D-function IGCM_sys_projectAccounting
1662#D-* Purpose: store project accounting information in a file
1663#D-* Examples:
1664#D-
1665function IGCM_sys_projectAccounting {
1666  IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1667  if ( $DEBUG_sys ) ; then
1668    echo "IGCM_sys_check_quota"
1669  fi
1670#need to implemented on Jean Zay
1671  touch $1
1672#  ssh irene191 /usr/bin/ccc_myproject > $1
1673
1674  IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1675}
1676
1677#D-#==================================================
1678#D-function IGCM_sys_getJobSchedulerID
1679#D-* Purpose: Get the job ID during execution
1680#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1681#D-
1682function IGCM_sys_getJobSchedulerID {
1683  IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1684  if ( $DEBUG_sys ) ; then
1685    echo "IGCM_sys_getJobSchedulerID"
1686  fi
1687
1688  eval ${1}=${SLURM_JOBID}
1689
1690  IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1691}
1692
1693#D-#==================================================
1694#D-function IGCM_sys_GetJobID
1695#D-* Purpose: Get the job ID from the JobName
1696#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1697#D-
1698function IGCM_sys_GetJobID {
1699  IGCM_debug_PushStack "IGCM_sys_GetJobID"
1700  if ( $DEBUG_sys ) ; then
1701    echo "IGCM_sys_GetJobID"
1702  fi
1703
1704  ID=$( squeue -u $user | grep Job_${config_UserChoices_JobName} | gawk ' { print $1}' )
1705
1706
1707  eval ${3}=${ID}
1708  IGCM_debug_PopStack "IGCM_sys_GetJobID"
1709}
1710
1711#D-#==================================================
1712#D-function IGCM_sys_CountJobInQueue
1713#D-* Purpose: Count number of users job
1714#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1715#D-
1716function IGCM_sys_CountJobInQueue {
1717  IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1718  if ( $DEBUG_sys ) ; then
1719    echo "IGCM_sys_CountJobInQueue"
1720  fi
1721
1722  # With -f option, the full job name is given in the last column
1723  NbRun=$(squeue -u $user | \
1724      grep -v JOBID | gawk 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1725
1726  eval ${2}=${NbRun}
1727
1728  IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1729}
1730
1731#D-#==================================================
1732#D-function IGCM_sys_ListJobInQueue
1733#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1734#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1735#D-
1736function IGCM_sys_ListJobInQueue {
1737  IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1738  if ( $DEBUG_sys ) ; then
1739    echo "IGCM_sys_ListJobInQueue"
1740  fi
1741
1742# to be implemented on Jean Zay
1743 set -A JobList $( squeue -u $user | gawk '{print $3}' | grep -v NAME)
1744#  # With -f option, the full job name is given in the last column
1745#  set -A JobList $( ccc_mstat -f | gawk -v User=$1             \
1746#                                        '( $2  == User      && \
1747#                                           $NF != /TS/      && \
1748#                                           $NF !~ /PACK/    && \
1749#                                           $NF !~ /REBUILD/ && \
1750#                                           $NF !~ /pack/ )     \
1751#                                         { print $NF }' | sed -e "s/\(.*\)\.[0-9]*/\1/" )
1752#
1753  eval set -A ${2} ${JobList[*]}
1754
1755  IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1756}
1757
1758#D-#==================================================
1759#D-function IGCM_sys_atlas
1760#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1761#D-* Examples:
1762#D-
1763function IGCM_sys_atlas {
1764  IGCM_debug_PushStack "IGCM_sys_atlas" $@
1765  if ( $DEBUG_sys ) ; then
1766    echo "IGCM_sys_atlas :" $@
1767  fi
1768
1769  typeset status
1770# To be implemented on Jean Zay
1771#
1772#  \ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1773#  status=$?
1774#  if [ ${status} -gt 0 ] ; then
1775#    echo "IGCM_sys_atlas : error code ${status}"
1776#    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1777#    \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1778#    IGCM_debug_PopStack "IGCM_sys_atlas"
1779#    return 1
1780#  else
1781#    IGCM_debug_PopStack "IGCM_sys_atlas"
1782#    return 0
1783#  fi
1784#
1785  IGCM_debug_PopStack "IGCM_sys_atlas"
1786}
1787
1788#D-#==================================================
1789#D-function IGCM_sys_rebuild_nemo
1790#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1791#D-* Examples:
1792#D-
1793
1794function IGCM_sys_rebuild_nemo {
1795  IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1796  if ( $DEBUG_sys ) ; then
1797    echo "IGCM_sys_rebuild_nemo :" $@
1798  fi
1799
1800  /gpfswork/rech/psl/commun/Tools/rebuild_nemo/bin/rebuild_nemo ${1} ${2}
1801
1802  IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1803}
1804
Note: See TracBrowser for help on using the repository browser.