source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1624

Last change on this file since 1624 was 1624, checked in by jgipsl, 4 months ago

Added possibility to declare a restart file as optional which means that libIGCM want stop if the file is not found. Only a warning will be written.

For example, in the following case, if the routing_restart file is not available when starting the model or if it is not produced, libIGCM will not stop. But if the file sechiba_rest is not available, libIGCM will stop as usual.

[RestartFiles]
List=   (sechiba_rest_out.nc, sechiba_rest.nc, sechiba_rest_in.nc) ,\
        (routing_restart.nc, routing_restart.nc, routing_start.nc, OPTIONAL)
  • Read optional argument OPTIONAL on the line where the restart file is declared.
  • Each time when a restart file is not found for coping in or out, if the argument OPTIONAL is set, then just continue. If not set, the file is considered as MANDATORY and libIGCM will exist if it is missing as before.
  • on irene-amd and jean-zay, corrected a bug for the case in config.card DataProject?=DEFAULT


  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 60.3 KB
Line 
1#!/bin/ksh
2
3#**************************************************************
4# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6# $Revision::                                          $ Revision of last commit
7# $Author::                                            $ Author of last commit
8# $Date::                                              $ Date of last commit
9# IPSL (2006)
10#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11#
12#**************************************************************
13
14#=========================================================
15# The documentation of this file can be automatically generated
16# if you use the prefix #D- for comments to be extracted.
17# Extract with command: cat lib* | grep "^#D-" | cut -c "4-"
18#=========================================================
19
20#D-#==================================================
21#D-LibIGCM_sys for Jean-Zay
22#D-#==================================================
23#D-
24#D- This ksh library if a layer under some usefull
25#D-environment variables and shell commands.
26#D-All those definitions depend on host particularities.
27#D-It manages a stack mechanism and test validity of operations.
28#D-All function described bellow must be prefixed by IGCM_sys.
29
30#====================================================
31# libIGCM_sys PARAMETERS
32#====================================================
33
34#====================================================
35# set DEBUG_sys to true to output calls of function
36typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38#====================================================
39# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
41  typeset -r DRYRUN=0
42  typeset -r DRYRUN_DEBUG=4
43else
44  typeset -r DRYRUN=${DRYRUN:=0}
45  typeset -r DRYRUN_DEBUG=0
46fi
47
48# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
49# -------------------------------------------------------------------------------------
50# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
51# |          |  Cp/Exe/param/files |            |         |                           |
52# |          |  Chmod Qsub         |            |         |                           |
53# -------------------------------------------------------------------------------------
54# |    0     |       yes           |    yes     |  yes    |      yes                  |
55# -------------------------------------------------------------------------------------
56# |    1     |       yes           |    yes     |  yes    |      no                   |
57# -------------------------------------------------------------------------------------
58# |    2     |       yes           |    yes     |  no     |      no                   |
59# -------------------------------------------------------------------------------------
60# |    3     |       yes           |    no      |  no     |      no                   |
61# -------------------------------------------------------------------------------------
62# |    4     |       yes           |    yes     |  creation of RUN_DIR and Job_debug  |
63# -------------------------------------------------------------------------------------
64
65
66#=====================================================
67# Global Variables :
68#=====================================================
69# Language : "fr" or "en"
70typeset -r MYLANG="fr"
71
72#=====================================================
73# Host user names project maxCpuTime
74# $hostname ou hostname
75typeset HOST=${HOST:=$( hostname )}
76# $username ou whoami
77typeset LOGIN=${LOGIN:=$( whoami )}
78# $hostname of the MASTER job
79typeset MASTER=jeanzay
80# add default project set by ins_job
81typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} | cut -c "1-3" ) 
82### jobWarningDelay in seconds
83##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
84 
85#D-
86#D-#==================================================
87#D-Program used in libIGCM
88#D-#==================================================
89
90# Submit command
91typeset SUBMIT=${SUBMIT:=sbatch}
92# rsync with path
93typeset -r RSYNC=/usr/bin/rsync
94# RSYNC_opt args to rsync
95typeset -r RSYNC_opt="-va"
96# ie storage filesystem
97typeset -r STOREHOST=${MASTER}
98typeset -r REMOTE_RSYNC=/usr/bin/rsync
99
100#====================================================
101# Access to module command
102#====================================================
103# See ticket #348
104# Following path should be used according to assist IDRIS
105. /gpfslocalsup/spack_soft/environment-modules/current/init/ksh
106# Following old path gives error messages while loading modules but it still works
107#. /usr/share/Modules/init/ksh
108
109#====================================================
110# Set environment tools (ferret, nco, cdo, rebuild, ...)
111#====================================================
112if [ X${TaskType} = Xcomputing ] ; then
113  IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
114else
115  module purge > /dev/null 2>&1
116  . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
117
118  . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
119
120#  export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
121  export UVCDAT_ANONYMOUS_LOG=FALSE
122fi
123
124# FYI
125[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
126[ ! X${TaskType} = Xchecking ] && module list
127
128# For AddNoise (not yet AddPertu3DOCE)
129export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/AddNoise/bin
130
131# For rebuild
132export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
133
134#====================================================
135# Host specific DIRECTORIES
136#====================================================
137
138#====================================================
139#- MirrorlibIGCM for frontend
140typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
141
142#====================================================
143#- libIGCM_POST for frontend
144typeset -r libIGCM_POST=${libIGCM}
145
146#====================================================
147#- R_EXE   (==> BIN_DIR = ${MODIPSL}/bin )
148typeset -r R_EXE="${MODIPSL}/bin"
149
150#====================================================
151#- SUBMIT_DIR : submission dir
152typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
153
154#====================================================
155#- IN
156typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
157
158#====================================================
159#- RUN_DIR_PATH : Temporary working directory (=> TMP)
160typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
161
162#====================================================
163#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
164typeset -r OUTCOMMAND_PATH=/tmp
165
166#====================================================
167#- HOST_MPIRUN_COMMAND
168typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
169
170#====================================================
171#- Max number of arguments passed to nco operator or demigration command
172UNIX_MAX_LIMIT=360
173
174#====================================================
175#- set PackDefault to true
176PackDefault=true
177
178#====================================================
179#- Default number of MPI task for IPSL coupled model
180#- required for backward compatibility
181#-
182DEFAULT_NUM_PROC_OCE=5
183DEFAULT_NUM_PROC_CPL=1
184(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
185
186#====================================================
187#- Number of computing cores per node
188#-
189NB_CORE_PER_NODE=40
190
191#D-#==================================================
192#D-function IGCM_sys_defineArchives
193#D-* Purpose:
194#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
195#D-* Define ARCHIVE : Dedicated to large files
196#D-* Define STORAGE : Dedicated to small/medium files
197#D-* Define R_OUT   : Output tree located on ARCHIVE
198#D-* Define R_FIG   : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
199#D-* Define R_BUF   : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
200#D-* if SpaceName=TEST everything is stored on SCRATCH
201#D-* Examples:
202#D-
203function IGCM_sys_defineArchives {
204  IGCM_debug_PushStack "IGCM_sys_defineArchives"
205
206  # Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
207  if [ X${config_UserChoices_DataProject} = X ] || [ X${config_UserChoices_DataProject} = XDEFAULT ]; then
208    # The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
209    # Default option: Set DataProject according to the project used for submission
210    DataProject=${PROJECT}
211  else
212    # Use the project set by the variable DataProject in section UserChoices in config.card
213    IGCM_debug_Print 1 "Use DataProject as set in config.card : ${config_UserChoices_DataProject}"
214    DataProject=${config_UserChoices_DataProject}
215  fi
216  IGCM_debug_Print 1 "Data project for output to be used: ${DataProject}"
217
218  #====================================================
219  #- RUN_DIR_PATH : Temporary working directory (=> TMP)
220  #- DataProject = for is the dataproject for Training account at Idris
221  #- they are not store in rech but in another partition
222  if [ X${DataProject} = Xfor  ]; then
223      typeset -r RUN_DIR_PATH=$SCRATCH/RUN_DIR/${SLURM_JOBID}_${$}
224  else
225      typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/gpfsscratch/rech/${DataProject}/${LOGIN}/RUN_DIR/${SLURM_JOBID}_${$}}
226  fi
227
228  if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
229    #====================================================
230    #- ARCHIVE (dedicated to large files)
231    ARCHIVE=${config_UserChoices_ARCHIVE}
232  else
233      if [ X${DataProject} = Xfor  ]; then
234          ARCHIVE=$STORE
235      else
236          #====================================================
237          #- ARCHIVE (dedicated to large files)
238          ARCHIVE=/gpfsstore/rech/${DataProject}/${LOGIN}
239      fi
240  fi
241
242  if [ ! X${config_UserChoices_STORAGE} = X ]; then
243    #====================================================
244    #- STORAGE (dedicated to small/medium files)
245    STORAGE=${config_UserChoices_STORAGE}
246  else
247    #====================================================
248    #- STORAGE (dedicated to small/medium files)
249      if [ X${DataProject} = Xfor  ]; then
250          STORAGE=$WORK
251      else
252          STORAGE=/gpfswork/rech/${DataProject}/${LOGIN}
253      fi
254  fi
255
256  if [ X${config_UserChoices_SpaceName} = XTEST ]; then
257    #====================================================
258    #- R_OUT
259    if [ X${DataProject} = Xfor  ]; then
260        R_OUT=$SCRATCH/IGCM_OUT
261    else
262        R_OUT=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
263    fi
264    #====================================================
265    #- R_FIG (hosting figures : monitoring and atlas, and/or small files)
266    R_FIG=${R_OUT}
267
268    IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
269
270  else
271    #====================================================
272    #- R_OUT
273    R_OUT=${ARCHIVE}/IGCM_OUT
274
275    #====================================================
276    #- R_FIG (hosting figures : monitoring and atlas, and/or small files)
277    R_FIG=${STORAGE}/IGCM_OUT
278  fi
279
280  #====================================================
281  #- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
282  CMIP6_BUF=${STORAGE}/IGCM_OUT
283
284  #====================================================
285  #- R_BUF (ONLY FOR double copy an scratch)
286 
287  if [ X${DataProject} = Xfor  ]; then
288      R_BUF=$SCRATCH/IGCM_OUT
289  else
290      R_BUF=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
291  fi
292
293  IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
294  IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
295  IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
296
297  IGCM_debug_PopStack "IGCM_sys_defineArchives"
298}
299
300#D-#==================================================
301#D-function IGCM_sys_RshArchive
302#D-* Purpose: Archive rsh command
303#D-* Examples:
304#D-
305function IGCM_sys_RshArchive {
306  IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
307  /bin/ksh <<-EOF
308    ${@}
309EOF
310  status=$?
311  if [ ${status} -gt 0 ] ; then
312    IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
313    IGCM_debug_Exit "IGCM_sys_RshArchive"
314  fi
315  IGCM_debug_PopStack "IGCM_sys_RshArchive"
316}
317
318#D-#==================================================
319#D-function IGCM_sys_RshArchive_NoError
320#D-* Purpose: Archive rsh command, without error
321#D-*          used only in monitoring.job
322#D-* Examples:
323#D-
324function IGCM_sys_RshArchive_NoError {
325  IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
326  /bin/ksh <<-EOF
327    ${@} 2> /dev/null
328EOF
329  IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
330}
331
332#D-#==================================================
333#D-function IGCM_sys_MkdirArchive
334#D-* Purpose: Mkdir on Archive
335#D-* Examples:
336#D-
337function IGCM_sys_MkdirArchive {
338  IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
339  if ( $DEBUG_sys ) ; then
340    echo "IGCM_sys_MkdirArchive :" $@
341  fi
342  #- creation de repertoire sur le serveur fichier
343  if [ ! -d ${1} ]; then
344    \mkdir -p $1
345    status=$?
346
347    if [ ${status} -gt 0 ] ; then
348      IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
349      IGCM_debug_Exit "IGCM_sys_MkdirArchive"
350    fi
351  fi
352  IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
353}
354
355#D-#==================================================
356#D-function IGCM_sys_TestDirArchive
357#D-* Purpose: Test Directory that must exists on Archive
358#D-* Examples:
359#D-
360function IGCM_sys_TestDirArchive {
361  IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
362  if ( $DEBUG_sys ) ; then
363    echo "IGCM_sys_TestDirArchive :" $@
364  fi
365  typeset ExistFlag
366  ExistFlag=$( [ -d $1 ] && echo 0 || echo 1 )
367  IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
368  return ${ExistFlag}
369}
370
371#D-#==================================================
372#D-function IGCM_sys_IsFileArchived
373#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
374#D-* Examples:
375#D-
376function IGCM_sys_IsFileArchived {
377  IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
378  if ( $DEBUG_sys ) ; then
379    echo "IGCM_sys_IsFileArchived :" $@
380  fi
381  typeset IsArchivedFlag
382# To be implemented on Jean Zay
383#  IsArchivedFlag=$( [ X$( echo $@ | grep ^\/ccc\/store ) != X ] && echo 0 || echo 1 )
384  IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
385
386  return ${IsArchivedFlag}
387}
388
389#D-#==================================================
390#D-function IGCM_sys_TestFileArchive
391#D-* Purpose: Test file that must NOT EXISTS on Archive
392#D-* Examples:
393#D-
394function IGCM_sys_TestFileArchive {
395  IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
396  typeset ExistFlag
397  ExistFlag=$( [ -f $1 ] && echo 0 || echo 1 )
398  IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
399
400  return ${ExistFlag}
401}
402
403#D-#==================================================
404#D-function IGCM_sys_CountFileArchive
405#D-* Purpose: Count files on Archive filesystem
406#D-* Examples:
407#D-
408function IGCM_sys_CountFileArchive {
409  IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
410  ls ${@} 2>/dev/null | wc -l
411  if [ $? -gt 0 ] ; then
412    echo "IGCM_sys_CountFileArchive : erreur."
413  fi
414  IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
415}
416
417#D-#==================================================
418#D-function IGCM_sys_Tree
419#D-* Purpose: Tree directories with files on ${ARCHIVE}
420#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
421#D-
422function IGCM_sys_Tree {
423  IGCM_debug_PushStack "IGCM_sys_Tree" $@
424  if ( $DEBUG_sys ) ; then
425    echo "IGCM_sys_Tree :" $@
426  fi
427
428  \ls -lR ${@}
429
430  IGCM_debug_PopStack "IGCM_sys_Tree"
431}
432
433#D-#==================================================
434#D-function IGCM_sys_Qsub
435#D-* Purpose: Qsub new job
436#D-* Examples:
437#D-
438function IGCM_sys_Qsub {
439  IGCM_debug_PushStack "IGCM_sys_Qsub" $@
440  if ( $DEBUG_sys ) ; then
441    echo "IGCM_sys_Qsub :" $@
442  fi
443  typeset options status
444  options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
445
446  /usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
447  status=$?
448
449  cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
450  if [ ${status} -gt 0 ] ; then
451    IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
452    IGCM_debug_Exit "IGCM_sys_Qsub"
453  else
454    JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
455  fi
456  IGCM_debug_PopStack "IGCM_sys_Qsub"
457}
458
459#D-#==================================================
460#D-function IGCM_sys_QsubPost
461#D-* Purpose: Qsub new job on scalaire
462#D-* Examples:
463#D-
464function IGCM_sys_QsubPost {
465  IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
466
467  BAK_DIR=$PWD 
468  cd ${SUBMIT_DIR}
469
470  if ( $DEBUG_sys ) ; then
471    echo "IGCM_sys_QsubPost :" $@
472  fi
473  typeset options status
474  options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
475
476  sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
477  status=$?
478
479  cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
480  if [ ${status} -gt 0 ] ; then
481    IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
482    IGCM_debug_Exit "IGCM_sys_QsubPost"
483  else
484    JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
485  fi
486
487  cd ${BAK_DIR}
488
489  IGCM_debug_PopStack "IGCM_sys_QsubPost"
490}
491
492#D-*************************
493#D- File transfer functions
494#D-*************************
495#D-
496
497#D-#==================================================
498#D-function IGCM_sys_RmRunDir
499#D-* Purpose: rm tmpdir (dummy function most of the time batch
500#D-                      scheduler will do the job)
501#D-* Examples:
502#D-
503function IGCM_sys_RmRunDir {
504  IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
505  if ( $DEBUG_sys ) ; then
506    echo "IGCM_sys_RmRunDir :" $@
507  fi
508
509  typeset status
510
511  echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
512  \rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
513  status=$?
514
515  if [ ${status} -gt 0 ] ; then
516    IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
517    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
518    IGCM_debug_Exit "IGCM_sys_RmRunDir"
519  else
520    \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
521  fi
522  IGCM_debug_PopStack "IGCM_sys_RmRunDir"
523}
524
525#D-#==================================================
526#D-function IGCM_sys_Put_Dir
527#D-* Purpose: Copy a complete directory on $(ARCHIVE)
528#D-* Examples:
529#D-
530function IGCM_sys_Put_Dir {
531  IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
532  if ( $DEBUG_sys ) ; then
533    echo "IGCM_sys_Put_Dir :" $@
534  fi
535  if [ $DRYRUN = 0 ]; then
536    if [ ! -d ${1} ] ; then
537      echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
538      IGCM_debug_PopStack "IGCM_sys_Put_Dir"
539      return
540    fi
541
542    typeset status
543
544    # Only if we use rsync
545    #IGCM_sys_TestDirArchive $( dirname $2 )
546    #
547    #USUAL WAY
548    \cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
549    status=$?
550
551    if [ ${status} -gt 0 ] ; then
552      IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
553      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
554      IGCM_debug_Exit "IGCM_sys_Put_Dir"
555    else
556      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
557    fi
558  fi
559  IGCM_debug_PopStack "IGCM_sys_Put_Dir"
560}
561
562#D-#==================================================
563#D-function IGCM_sys_Get_Dir
564#D-* Purpose: Copy a complete directory from ${ARCHIVE}
565#D-* Examples:
566#D-
567function IGCM_sys_Get_Dir {
568  IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
569  if ( $DEBUG_sys ) ; then
570    echo "IGCM_sys_Get_Dir :" $@
571  fi
572  if [ $DRYRUN = 0 ]; then
573    typeset NB_ESSAI DELAI status i
574    # number of tentative
575    NB_ESSAI=3
576    # time delay between tentative
577    DELAI=2
578
579    #
580    # USUAL WAY
581    # add here command to demigrate all offline files to reduce time of this command
582    # actually on Jean Zay files are not archive on bands
583
584    i=0
585    while [ $i -lt $NB_ESSAI ] ; do
586      \cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
587      status=$?
588      if [ ${status} -gt 0 ] ; then
589        IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
590        IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
591        sleep $DELAI
592      else
593        break
594      fi
595      (( i = i + 1 ))
596    done
597
598    if [ ${status} -gt 0 ] ; then
599      IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
600      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
601      IGCM_debug_Exit "IGCM_sys_Get_Dir"
602    else
603      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
604    fi
605  fi
606  IGCM_debug_PopStack "IGCM_sys_Get_Dir"
607}
608
609#D-#==================================================
610#D-function IGCM_sys_Put_Rest
611#D-* Purpose: Put computied restarts on ${ARCHIVE}.
612#D-           File and target directory must exist.
613#D-* Examples:
614#D-
615function IGCM_sys_Put_Rest {
616  IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
617  if ( $DEBUG_sys ) ; then
618    echo "IGCM_sys_Put_Rest :" $@
619  fi
620  if [ $DRYRUN = 0 ]; then
621    if [ ! -f ${1} ] ; then
622        # The file is missing
623        if [ X${3} = X'OPTIONAL' ] ; then
624            echo "WARINING : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
625            IGCM_debug_Print 1 "IGCM_sys_Put_Rest: Optional restart file ${1} is missing, now continue for the next"
626        else
627            echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
628            IGCM_debug_Exit "IGCM_sys_Put_Rest"
629        fi
630
631    else
632      # The file exist
633
634    typeset status
635    #
636    # USUAL WAY
637    \cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
638    status=$?
639
640#       #RSYNC WITH NETWORK SSH CALL
641#       echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
642#       ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
643
644#       #RSYNC WITH NFS USE
645#       echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
646#       ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
647
648#       status=$?
649#       IGCM_sys_Rsync_out $status
650
651#       ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
652#       (( status=status+$? ))
653
654    if [ ${status} -gt 0 ] ; then
655      IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
656      [ -f ${1} ] && ls -l ${1}
657      [ -f ${2} ] && ls -l ${2}
658      [ -f ${2}/${1} ] && ls -l ${2}/${1}
659      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
660      IGCM_debug_Exit "IGCM_sys_Put_Rest"
661    else
662
663      if [ X${JobType} = XRUN ] ; then
664        [ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
665        [ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
666      fi
667
668      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
669    fi
670    fi
671  fi
672  IGCM_debug_PopStack "IGCM_sys_Put_Rest"
673}
674
675#D-#==================================================
676#D-function IGCM_sys_Put_Out
677#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
678#D-* Examples:
679#D-
680function IGCM_sys_Put_Out {
681  IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
682  if ( $DEBUG_sys ) ; then
683    echo "IGCM_sys_Put_Out :" $@
684  fi
685
686  typeset NB_ESSAI DELAI status i exist skip
687  typeset fileDeviceNumberInHex directoryDeviceNumberInHex
688
689  # number of tentative
690  NB_ESSAI=3
691  # time delay between tentative
692  DELAI=2
693
694  if [ $DRYRUN = 0 ]; then
695    if [ ! -f ${1} ] ; then
696      echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
697      IGCM_debug_PopStack "IGCM_sys_Put_Out"
698      return 1
699    fi
700    #
701    IGCM_sys_MkdirArchive $( dirname $2 )
702    #
703    exist=false
704    skip=false
705    if [ -f $2 ] ; then
706      IGCM_debug_Print 1 "$2 already exist"
707    # add here command to demigrate all offline files to reduce time of this command
708    # actually on Jean Zay files are not archive on bands
709
710      exist=true
711      if [ "X$( diff $1 $2 )" = X ] ; then
712        IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
713        skip=true
714      else
715        IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
716        skip=false
717      fi
718    fi
719    #
720    if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
721      IGCM_sys_Chmod u+w $2
722    fi
723
724    if [ X${skip} = Xfalse ] ; then
725      i=0
726      while [ $i -lt $NB_ESSAI ] ; do
727        # Identify file system
728        fileDeviceNumberInHex=$( stat -c %d $1 )
729        status=$?
730        if [ ${status} -gt 0 ] ; then
731          IGCM_debug_Exit "IGCM_sys_Put_Out"
732        fi
733        # Identify file system
734        directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
735        status=$?
736        if [ ${status} -gt 0 ] ; then
737          IGCM_debug_Exit "IGCM_sys_Put_Out"
738        fi
739
740        if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] || [ X$3 = XNOMOVE ] ) ; then
741          # They are not on the same device. USUAL WAY
742          \cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
743          status=$?
744        else
745          # They are on the same device. NOT SO USUAL WAY
746          \mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
747          status=$?
748        fi
749        if [ ${status} -gt 0 ]; then
750          IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
751          IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
752          [ -f ${1} ] && ls -l ${1}
753          [ -f ${2} ] && ls -l ${2}
754          [ -f ${2}/${1} ] && ls -l ${2}/${1}
755          sleep $DELAI
756        else
757          break
758        fi
759        (( i = i + 1 ))
760      done
761    fi
762
763#       #RSYNC WITH NETWORK SSH CALL
764#       echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
765#       ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
766
767#       #RSYNC WITH NFS USE
768#       echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
769#       ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
770
771#       status=$?
772#       IGCM_sys_Rsync_out $status
773
774#       ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
775#       (( status=status+$? ))
776
777    if [ ${status} -gt 0 ] ; then
778      IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
779      [ -f ${1} ] && ls -l ${1}
780      [ -f ${2} ] && ls -l ${2}
781      [ -f ${2}/${1} ] && ls -l ${2}/${1}
782      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
783      IGCM_debug_Exit "IGCM_sys_Put_Out"
784    else
785
786      if [ X${JobType} = XRUN ] ; then
787        if [ X${3} = X ] ; then
788          [ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
789          [ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
790        fi
791      fi
792
793      \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
794    fi
795  fi
796  IGCM_debug_PopStack "IGCM_sys_Put_Out"
797  return 0
798}
799
800#D-#==================================================
801#D-function IGCM_sys_Get
802#D-* Purpose: Get a file from ${ARCHIVE}
803#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
804#D-            IGCM_sys_Get /l Array_contain_myfiles /destpath/
805function IGCM_sys_Get {
806  IGCM_debug_PushStack "IGCM_sys_Get" $@
807
808  typeset DEST dm_liste target file_work
809  typeset NB_ESSAI DELAI status i
810
811  if ( $DEBUG_sys ) ; then
812    echo "IGCM_sys_Get :" $@
813  fi
814
815  # number of tentative
816  NB_ESSAI=3
817  # time delay between tentative
818  DELAI=2
819
820  if [ $DRYRUN -le 2 ]; then
821    if [ X${1} = X'/l' ] ; then
822      eval set +A dm_liste \${${2}}
823    else
824      eval set +A dm_liste ${1}
825    fi
826    eval DEST=\${${#}}
827    # add here command to demigrate all offline files to reduce time of this command
828    # actually on Jean Zay files are not archive on bands
829
830    status=$?
831    if [ ${status} -gt 0 ] ; then
832      echo "WARNING IGCM_sys_Get : error code ${status}"
833      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
834      echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
835    fi
836
837    #if [ ${status} -gt 0 ] ; then
838    #  if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
839    #    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
840    #    echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
841    #    sleep 30
842    #    echo "We try another time"
843    ##    dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
844    #    ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
845    #    status=$?
846    #    if [ ${status} -gt 0 ] ; then
847    #      echo "ERROR IGCM_sys_Get : again demigration error :"
848    #      cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
849    #      IGCM_debug_Exit "IGCM_sys_Get"
850    #    fi
851    #  else
852    #    echo "ERROR IGCM_sys_Get : demigration error :"
853    #    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
854    #    IGCM_debug_Exit "IGCM_sys_Get"
855    #  fi
856    #fi
857
858    #   #RSYNC WITH NETWORK SSH CALL
859    #   echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
860    #   ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
861
862    #   #RSYNC WITH NFS USE
863    #   echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
864    #   ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
865
866    #   status=$?
867    #   IGCM_sys_Rsync_out $status
868
869    #   ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
870    #   (( status=status+$? ))
871
872    #USUAL WAY
873    if [ X${1} = X'/l' ] ; then
874      for target in ${dm_liste[*]} ; do
875        local_file=$( basename ${target} )
876        # test if the target file is present before the loop
877        IGCM_sys_TestFileArchive ${target}
878        status=$?
879        if [ ${status} -gt 0 ] ; then
880          echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
881          IGCM_debug_Exit "IGCM_sys_Get"
882        else
883          i=0
884          while [ $i -lt $NB_ESSAI ] ; do
885            #if [ X${DoLink} = Xtrue ] ; then
886            #  \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
887            #  status=$?
888            #  else
889            #  \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
890            #  status=$?
891            #fi
892            \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
893            status=$?
894            if [ ${status} -gt 0 ]; then
895              IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
896              IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
897              sleep $DELAI
898            else
899              break
900            fi
901            (( i = i + 1 ))
902          done
903          if [ ${status} -gt 0 ] ; then
904            echo "IGCM_sys_Get : error"
905            cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
906            \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
907            IGCM_debug_Exit "IGCM_sys_Get"
908          else
909            \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
910          fi
911        fi
912      done
913    else
914      i=0
915      while [ $i -lt $NB_ESSAI ] ; do
916        \cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
917        status=$?
918        if [ ${status} -gt 0 ]; then
919          IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
920          IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
921          sleep $DELAI
922        else
923          break
924        fi
925        (( i = i + 1 ))
926      done
927      if [ ${status} -gt 0 ] ; then
928        echo "IGCM_sys_Get : error"
929        cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
930        \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
931        IGCM_debug_Exit "IGCM_sys_Get"
932      else
933        \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
934      fi
935    fi
936  fi
937  IGCM_debug_PopStack "IGCM_sys_Get"
938}
939
940#D-#==================================================
941#D-function IGCM_sys_GetDate_Monitoring
942#D-* Purpose: get the last year for which the monitoring has been computed
943#D-* Examples:
944#D-
945function IGCM_sys_GetDate_Monitoring {
946  IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
947  if ( $DEBUG_sys ) ; then
948    echo "IGCM_sys_GetDate_Monitoring :" $@
949  fi
950
951  eval ${2}=$( cdo showyear ${1} 2> /dev/null | gawk '{ print $NF }' )
952
953  IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
954}
955
956#D-#==================================================
957#D-function IGCM_sys_Dods_Rm
958#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
959#D-* Examples:
960#D-
961function IGCM_sys_Dods_Rm {
962  if ( $DEBUG_sys ) ; then
963    echo "IGCM_sys_Dods_Rm :" $@
964  fi
965  typeset status
966  if [ $DRYRUN = 0 ]; then
967
968#    if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
969#      echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
970#      echo "Nothing has been done."
971#      return
972#    fi
973
974    if [ "$#" -eq 1 ]; then
975      thredds_rm ${R_DODS}/${1}
976      status=$?
977    else
978        thredds_rm ${R_DODS}
979      status=$?
980    fi
981
982#    if [ ${status} -gt 0 ] ; then
983#      echo "IGCM_sys_Dods_Rm : error."
984#      cat out_dods_rm
985#      IGCM_debug_Exit "IGCM_sys_Dods_Rm"
986#    else
987#      rm out_dods_rm
988#    fi
989
990  fi
991  return $status
992}
993
994#D-#==================================================
995#D-function IGCM_sys_Dods_Cp
996#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
997#D-* Examples:
998#D-
999function IGCM_sys_Dods_Cp {
1000  if ( $DEBUG_sys ) ; then
1001    echo "IGCM_sys_Dods_Cp :" $@
1002  fi
1003  typeset status
1004  if [ $DRYRUN = 0 ]; then
1005
1006#    if [ ! -d ${R_SAVE}/${1} ] ; then
1007#      echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
1008#      echo "Nothing has been done."
1009#      return
1010#    fi
1011
1012
1013      thredds_cp ${1} ./${R_DODS}/.
1014
1015    status=$?
1016
1017#       if [ ${status} -gt 0 ] ; then
1018#           echo "IGCM_sys_Dods_Cp : error."
1019#           cat out_dods_cp
1020#           IGCM_debug_Exit "IGCM_sys_Dods_Cp"
1021#       else
1022#           rm out_dods_cp
1023#       fi
1024
1025  fi
1026  return $status
1027}
1028
1029#D-#==================================================
1030#D-function IGCM_sys_Put_Dods
1031#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1032#D-* Examples:
1033#D-
1034function IGCM_sys_Put_Dods {
1035  IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1036  if ( $DEBUG_sys ) ; then
1037    echo "IGCM_sys_Put_Dods :" $@
1038  fi
1039  typeset status
1040  if [ $DRYRUN = 0 ]; then
1041    if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1042      echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1043      echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1044      echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1045      IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1046      return
1047    fi
1048
1049    if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1050      echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1051      echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1052      echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1053      IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1054      return
1055    fi
1056    #
1057    if [ -d ${R_SAVE}/${1} ] ; then
1058      cd ${R_SAVE}
1059    elif [ -d ${R_FIGR}/${1} ] ; then
1060      cd ${R_FIGR}
1061    fi
1062
1063    IGCM_sys_Dods_Rm ${1}
1064    IGCM_sys_Dods_Cp ${1}
1065    status=0
1066
1067    if [ ${status} -gt 0 ] ; then
1068      echo "IGCM_sys_Put_Dods : error."
1069      IGCM_debug_Exit "IGCM_sys_Put_Dods"
1070    fi
1071  fi
1072  IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1073}
1074
1075##############################################################
1076# REBUILD OPERATOR
1077
1078#D-#==================================================
1079#D-function IGCM_sys_sync
1080#D-* Purpose: flush buffer on disk
1081#D-* Examples:
1082#D-
1083function IGCM_sys_sync {
1084  IGCM_debug_PushStack "IGCM_sys_sync" $@
1085  if ( $DEBUG_sys ) ; then
1086    echo "IGCM_sys_sync :" $@
1087  fi
1088
1089  /bin/sync
1090
1091  IGCM_debug_PopStack "IGCM_sys_sync"
1092}
1093
1094############################################################
1095# Activate Running Environnment Variables
1096
1097#D-#==================================================
1098#D-function IGCM_sys_activ_variables
1099#D-* Purpose: set environement variables prior to execution
1100#D-* Examples:
1101#D-
1102function IGCM_sys_activ_variables {
1103  IGCM_debug_PushStack "IGCM_sys_activ_variables"
1104  if ( $DEBUG_sys ) ; then
1105    echo "IGCM_sys_activ_variables"
1106  fi
1107
1108# --------------------------------------------------------------------
1109#D- MODULE specifications
1110# --------------------------------------------------------------------
1111  # Source the file EnvFile containing all module load needed to run the model.
1112  # EnvFile can be specified in config.card. If this is the case and if the file exists,
1113  # it will be used. Otherwise a default file will be used.
1114  if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1115      # EnvFile is set in config.card and the file exists
1116      IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1117      EnvFile=${config_UserChoices_EnvFile}
1118  else
1119      IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1120      module purge > /dev/null 2>&1
1121      EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1122  fi
1123  IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :" 
1124  IGCM_debug_Print 1 ${EnvFile}
1125  . ${EnvFile}
1126
1127  IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1128  module list
1129
1130# --------------------------------------------------------------------
1131#D- MPI specifications
1132# --------------------------------------------------------------------
1133
1134# --------------------------------------------------------------------
1135#D- Other specifications
1136# --------------------------------------------------------------------
1137
1138  ulimit -s unlimited
1139  if [ ${executionType} -eq 2 ] ; then
1140    export SLURM_HOSTFILE=./hostlist
1141  fi
1142
1143
1144  IGCM_debug_PopStack "IGCM_sys_activ_variables"
1145}
1146
1147############################################################
1148# Desactivate Running Environnment Variables
1149
1150#D-#==================================================
1151#D-function IGCM_sys_desactiv_variables
1152#D-* Purpose: unset environement variables after execution
1153#D-* Examples:
1154#D-
1155function IGCM_sys_desactiv_variables {
1156  IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1157  if ( $DEBUG_sys ) ; then
1158    echo "IGCM_sys_desactiv_variables"
1159  fi
1160# --------------------------------------------------------------------
1161#D- MPI specifications
1162# --------------------------------------------------------------------
1163  if [ ${executionType} -eq 2 ] ; then
1164    unset SLURM_HOSTFILE
1165  fi
1166# --------------------------------------------------------------------
1167#D- Other specifications
1168# --------------------------------------------------------------------
1169
1170  IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1171}
1172
1173############################################################
1174# Update job headers to be used by the scheduler
1175
1176#D-#==================================================
1177#D-function IGCM_sys_updateHeaders
1178#D-* Purpose: Update job headers to be used by the scheduler
1179#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1180#D-
1181function IGCM_sys_updateHeaders {
1182  IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1183  if ( $DEBUG_sys ) ; then
1184    echo "IGCM_sys_updateHeaders"
1185  fi
1186  typeset file
1187  file=$1
1188
1189  if [ ${executionType} -eq 1 ] ; then
1190    # MPMD + MPI
1191    sed -e "/::openMPthreads::/d"                  \
1192        -e "s/::JobNumProcTot::/${coreNumber}/"    \
1193      ${file} > ${file}.tmp
1194
1195  elif [ ${executionType} -eq 2 ] ; then
1196    # MPMD + MPI + OMP
1197      (( nodeNumber = coreNumber / NB_CORE_PER_NODE ))
1198      [ $(( ${coreNumber} % ${NB_CORE_PER_NODE} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
1199      sed -e "/::openMPthreads::/d"                 \
1200        -e "s/::JobNumProcTot::/${mpiTasks}/"       \
1201        -e "/ntasks/i\#SBATCH --nodes=${nodeNumber}"\
1202        -e "/ntasks/i\#SBATCH --exclusive"          \
1203         ${file} > ${file}.tmp
1204  elif [ ${executionType} -eq 3 ] ; then
1205    # SPMD + MPI/OMP
1206    sed -e "s/::openMPthreads::/${openMPthreads}/" \
1207        -e "s/::JobNumProcTot::/${mpiTasks}/"      \
1208      ${file} > ${file}.tmp
1209
1210  elif [ ${executionType} -eq 4 ] ; then
1211    # SPMD + MPI only
1212    sed -e "s/::JobNumProcTot::/${mpiTasks}/"      \
1213        -e "/::openMPthreads::/d"                  \
1214      ${file} > ${file}.tmp
1215
1216  elif [ ${executionType} -eq 5 ] ; then
1217    # SPMD + OMP only
1218    sed -e "s/::openMPthreads::/${openMPthreads}/" \
1219        -e "/::JobNumProcTot::/d"                  \
1220      ${file} > ${file}.tmp
1221
1222  elif [ ${executionType} -eq 6 ] ; then
1223    # SEQUENTIAL THEN
1224    sed -e "s/::JobNumProcTot::/1/"                \
1225        -e "/::openMPthreads::/d"                  \
1226      ${file} > ${file}.tmp
1227
1228  fi
1229
1230  IGCM_sys_Mv ${file}.tmp ${file}
1231
1232  IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1233}
1234
1235############################################################
1236# Build MPI/OMP scripts run file (dummy function)
1237
1238#D-#==================================================
1239#D-function IGCM_sys_build_run_file
1240#D-* Purpose: build run file (deprecated)
1241#D-* Examples:
1242#D-
1243function IGCM_sys_build_run_file {
1244  IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1245}
1246
1247############################################################
1248# Build MPI/OMP scripts
1249
1250#D-#==================================================
1251#D-function IGCM_sys_build_execution_scripts
1252#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1253#D-* Examples:
1254#D-
1255function IGCM_sys_build_execution_scripts
1256{
1257  IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1258  if ( $DEBUG_sys ) ; then
1259    echo "IGCM_sys_build_execution_scripts " $@
1260  fi
1261  IGCM_debug_Print 1 "executionType= ${executionType}"
1262  EXECUTION=${HOST_MPIRUN_COMMAND}
1263
1264  if ( ${OK_PARA_MPMD} ) ; then
1265
1266    # MPMD mode
1267    # 1 MPI only : executionType=1
1268    # 2 MPI/OpenMP : executionType=2
1269
1270    if [ -f run_file ] ; then
1271      IGCM_sys_Rm -f run_file
1272    fi
1273    touch run_file
1274
1275    # case 1 : Only MPI (MPMD)
1276    if  ( ! ${OK_PARA_OMP} ) ; then
1277      # Build run_file
1278      current_core=0
1279      # First loop on the components for the coupler ie oasis (only if oasis3)
1280      # the coupler ie oasis3 must be the first one
1281      for comp in ${config_ListOfComponents[*]} ; do
1282
1283        eval ExeNameIn=\${config_Executable_${comp}[0]}
1284        eval ExeNameOut=\${config_Executable_${comp}[1]}
1285
1286        # Only if the component has an executable
1287        if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1288
1289          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1290          (( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
1291          echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
1292          (( current_core = ${end_core} + 1 ))
1293        fi
1294      done
1295
1296      EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
1297
1298      IGCM_sys_Chmod u+x run_file
1299      if ( $DEBUG_sys ) ; then
1300        echo "run_file contains : "
1301        cat run_file
1302      fi
1303
1304    else 
1305
1306    # 2 MPI/OpenMP : executionType=2
1307
1308    # MPI-OpenMP (MPMD)
1309    # export SLURM_HOSTFILE=./hostlist
1310    # srun --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file
1311    # example of  hostlist file :
1312    # r3i3n33
1313    # r3i3n33
1314    # ...
1315    # example of run_file :
1316    # 0-70 ./prog_lmdz.x.sh %o %t
1317    # 71-430 ./prog_opa.xx.sh %o %t
1318    # 431-431 ./prog_xios.x.sh %o %t
1319    # examples of prog_file :
1320    # prog_lmdz.x.sh :
1321    # (( init = 0 + $1 ))
1322    # (( index = init * 10 ))
1323    # (( slot = index % 40 ))
1324    # taskset -c $slot-$((slot + 10 - 1)) ./script_lmdz.x.ksh
1325    # that will become
1326    # taskset -c 0-9 ./script_lmdz.x.ksh
1327    # ...
1328    # with script_lmdz.x.ksh
1329    # export OMP_STACKSIZE=3g
1330    # export OMP_PLACES=cores
1331    # export OMP_NUM_THREADS=10
1332    # ./lmdz.x > out_lmdz.x.out.${SLURM_PROCID} 2>out_lmdz.x.err.${SLURM_PROCID}
1333
1334    #  Hosts treatment
1335    _bkIFS=$IFS;
1336        IFS=$'\n'; set -f
1337        listnodes=($(< <( scontrol show hostnames $SLURM_JOB_NODELIST )))
1338        IFS=$_bkIFS; set +f
1339        rm -f hostlist
1340
1341     # Loop on the components to build run_file and script_exec files
1342        rank=0
1343        current_core=0
1344        current_core_mpi=0
1345
1346      for comp in ${config_ListOfComponents[*]} ; do
1347
1348        eval ExeNameIn=\${config_Executable_${comp}[0]}
1349        eval ExeNameOut=\${config_Executable_${comp}[1]}
1350
1351        # Not possible if oasis has an executable (i.e old version of oasis3)
1352        if ( [ "X${ExeNameOut}" != X\"\" ] &&  [ "X${comp}" = "XCPL" ] ) ; then
1353          IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1354          IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1355          IGCM_debug_Verif_Exit
1356        fi
1357
1358        # Only if we really have an executable for the component :
1359        if [ "X${ExeNameOut}" != X\"\" ] ; then
1360
1361          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1362          eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1363          eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1364     
1365
1366          # Build script files
1367
1368          echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1369          echo ""  >> script_${ExeNameOut}.ksh
1370          if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1371          # Check if the number of threads is correct
1372              case ${comp_proc_omp_loc} in
1373                  2|4|5|10|20)
1374                      IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1375                      ;;
1376                  *)
1377                      IGCM_debug_Exit "ERROR with OMP parameters !"
1378                      IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1379                      IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1380                      IGCM_debug_Verif_Exit
1381                      ;;
1382              esac
1383            echo "export OMP_STACKSIZE=3g"  >> script_${ExeNameOut}.ksh
1384            echo "export OMP_PLACES=cores"  >> script_${ExeNameOut}.ksh
1385            echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}"  >> script_${ExeNameOut}.ksh
1386
1387          fi
1388
1389          # to have out/err per process on different files
1390          echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1391
1392          IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1393
1394        # Build run_file
1395        # Only if the component has an executable
1396        if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
1397
1398          eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1399          (( end_core = ${current_core_mpi} + ${comp_proc_mpi_loc} - 1 ))
1400          echo "${current_core_mpi}-${end_core} ./prog_${ExeNameOut}.sh %o %t" >> run_file
1401          (( current_core_mpi = ${end_core} + 1 ))
1402        fi
1403
1404          if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1405              (( offset_comp_proc_loc =  NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1406          else
1407              (( offset_comp_proc_loc =  comp_proc_omp_loc ))
1408          fi
1409
1410          # Build configuration file
1411 
1412              echo "#!/bin/sh" > prog_${ExeNameOut}.sh
1413          echo "(( init = $current_core + \$1 ))" >> prog_${ExeNameOut}.sh
1414          echo "(( index = init * $comp_proc_omp_loc ))" >> prog_${ExeNameOut}.sh
1415          echo "(( slot = index % 40 ))" >> prog_${ExeNameOut}.sh
1416              echo "echo ${ExeNameOut} taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1))" >> prog_${ExeNameOut}.sh
1417          echo "taskset -c \$slot"-"\$((slot + $comp_proc_omp_loc - 1)) ./script_${ExeNameOut}.ksh" >> prog_${ExeNameOut}.sh
1418
1419          IGCM_sys_Chmod u+x prog_${ExeNameOut}.sh
1420
1421        # Build hostlist file
1422
1423          for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1424             (( index_host = current_core / NB_CORE_PER_NODE ))
1425             host_value=${listnodes[${index_host}]}
1426             echo "$host_value" >> hostlist
1427             if [ ${DRYRUN_DEBUG} = 4 ] ; then
1428                 echo "node_${index_host}_X" >> hostlist_template
1429             fi
1430             (( current_core = current_core + offset_comp_proc_loc ))
1431          done
1432        fi
1433      done
1434
1435      ## variable added to stop after 60s instead of 600s by default.
1436      ## This is used when no error comes from executables and when something stopped an executable without notice.
1437      export SLURM_WAIT=60
1438
1439      EXECUTION="${HOST_MPIRUN_COMMAND} --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file"
1440
1441      IGCM_sys_Chmod u+x run_file
1442      if ( $DEBUG_sys ) ; then
1443        echo "run_file contains : "
1444        cat run_file
1445      fi
1446
1447    fi # if ${OK_PARA_MPMD}
1448
1449  else
1450  # Only one executable (SPMD mode):  executionType=3, 4, 5 and 6
1451
1452    for comp in ${config_ListOfComponents[*]} ; do
1453
1454      # Only if we really have an executable for the component :
1455      eval ExeNameOut=\${config_Executable_${comp}[1]}
1456      if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1457
1458        # Build script files
1459
1460        echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1461        echo ""  >> script_${ExeNameOut}.ksh
1462        IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1463
1464        if ( ${OK_PARA_OMP} ) ; then
1465            eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1466            # Check if the number of threads is correct
1467            case ${comp_proc_omp_loc} in
1468                2|4|5|10|20)
1469                    IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1470                    ;;
1471                *)
1472                    IGCM_debug_Exit "ERROR with OMP parameters !"
1473                    IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1474                    IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
1475                    IGCM_debug_Verif_Exit
1476                    ;;
1477            esac
1478            echo ""  >> script_${ExeNameOut}.ksh
1479            echo "export OMP_STACKSIZE=3g"  >> script_${ExeNameOut}.ksh
1480            echo "export OMP_PLACES=cores"  >> script_${ExeNameOut}.ksh
1481            echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1482        fi
1483
1484        eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1485       
1486        # To have out/err per process on different files
1487        echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}"  >> script_${ExeNameOut}.ksh
1488        EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
1489
1490        IGCM_debug_Print 1 "sys Jean-Zay : script_${ExeNameOut}.ksh contains"
1491        cat script_${ExeNameOut}.ksh
1492
1493      fi
1494
1495    done
1496
1497  fi # ${OK_PARA_MPMD}
1498
1499  IGCM_debug_Print 1 "sys Jean-Zay : execution command is "
1500  IGCM_debug_Print 1 "$EXECUTION"
1501
1502  IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1503}
1504
1505#D-#==================================================
1506#D-function IGCM_sys_check_path
1507#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1508#D-* do not point to an important use directory. Stop immediately in that case.
1509#D-* Examples:
1510#D-
1511function IGCM_sys_check_path {
1512  IGCM_debug_PushStack "IGCM_sys_check_path"
1513  if ( $DEBUG_sys ) ; then
1514    echo "IGCM_sys_check_path"
1515  fi
1516
1517  if ( [ X${RUN_DIR_PATH} = X${HOME} ] || [ X${RUN_DIR_PATH} = X${WORK} ] || [ X${RUN_DIR_PATH} = X${SCRATCH} ] || [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1518    IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1519    IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1520    IGCM_debug_Exit "This will stop the job"
1521  elif ( [ X${RUN_DIR_PATH} = X${HOME} ] || [ X${RUN_DIR_PATH} = X/gpfswork/rech/${DataProject}/${LOGIN} ] || [ X${RUN_DIR_PATH} = X/gpfsscratch/rech/${DataProject}/${LOGIN} ] || [ X${RUN_DIR_PATH} = X/gpfsstore/rech/${DataProject}/${LOGIN} ] ) ; then
1522    IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1523    IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1524    IGCM_debug_Exit "This will stop the job"
1525  fi
1526  IGCM_debug_PopStack "IGCM_sys_check_path"
1527}
1528
1529#D-#==================================================
1530#D-function IGCM_sys_check_quota
1531#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1532#D-* Examples:
1533#D-
1534function IGCM_sys_check_quota {
1535  IGCM_debug_PushStack "IGCM_sys_check_quota"
1536  echo "IGCM_sys_check_quota is desactivated on JeanZay because there is actually no way to check scratch quota on this computing center (2019.11.19)"
1537#  if ( $DEBUG_sys ) ; then
1538#    echo "IGCM_sys_check_quota"
1539#  fi
1540#  # Limit of quota (in %)
1541#  limit_quota=90
1542#
1543#  # Check of the volume
1544#  volume_quota=$(ccc_quota | grep ' scratch' | gawk '{print $2}')
1545#  volume_avail=$(ccc_quota | grep ' scratch' | gawk '{print $3}')
1546#
1547#  if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1548#
1549#    unit_avail=${volume_avail: -1}
1550#    unit_quota=${volume_quota: -1}
1551#
1552#    if [ "${unit_quota}" = "*" ] ; then
1553#      IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1554#      IGCM_debug_Print 1 "More than 100% of your quota is used"
1555#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1556#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1557#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1558#      IGCM_debug_Verif_Exit
1559#    fi
1560#
1561#    temp_avail=${volume_avail%%${volume_avail: -1}*}
1562#    temp_quota=${volume_quota%%${volume_quota: -1}*}
1563#
1564#    if [ ! ${unit_avail} = ${unit_quota} ] ; then
1565#
1566#    # Convertion
1567#      if [ ${volume_avail: -1} = "T" ] ; then
1568#        (( temp_avail = temp_avail * 1000000000000 ))
1569#      elif [ ${volume_avail: -1} = "G" ] ; then
1570#        (( temp_avail = temp_avail * 1000000000 ))
1571#      elif [ ${volume_avail: -1} = "M" ] ; then
1572#        (( temp_avail = temp_avail * 1000000 ))
1573#      elif [ ${volume_avail: -1} = "k" ] ; then
1574#        (( temp_avail = temp_avail * 1000 ))
1575#      else
1576#        (( temp_avail = volume_avail ))
1577#      fi
1578#      if [ ${volume_quota: -1} = "T" ] ; then
1579#        (( temp_quota = temp_quota * 1000000000000 ))
1580#      elif [ ${volume_quota: -1} = "G" ] ; then
1581#        (( temp_quota = temp_quota * 1000000000 ))
1582#      elif [ ${volume_quota: -1} = "M" ] ; then
1583#        (( temp_quota = temp_quota * 1000000 ))
1584#      elif [ ${volume_quota: -1} = "k" ] ; then
1585#        (( temp_quota = temp_quota * 1000 ))
1586#      else
1587#        (( temp_quota = volume_quota ))
1588#      fi
1589#    fi
1590#
1591#    quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" | bc)
1592##    echo "volume ratio is " $quota_volume
1593#
1594#    if [ ${quota_volume} -ge ${limit_quota} ] ; then
1595#      IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1596#      IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1597#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1598#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1599#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1600#      IGCM_debug_Verif_Exit
1601#    fi
1602#
1603#  fi
1604#
1605## Check of the number of inodes
1606#
1607#  inode_quota=$(ccc_quota | grep ' scratch' | gawk '{print $6}')
1608#  inode_avail=$(ccc_quota | grep ' scratch' | gawk '{print $7}')
1609#
1610#  if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1611#
1612#    unit_avail=${inode_avail: -1}
1613#    unit_quota=${inode_quota: -1}
1614#
1615#    if [ "${unit_quota}" = "*" ] ; then
1616#      IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1617#      IGCM_debug_Print 1 "More than 100% of your quota is used"
1618#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1619#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1620#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1621#      IGCM_debug_Verif_Exit
1622#    fi
1623#
1624#    temp_avail=${inode_avail%%${inode_avail: -1}*}
1625#    temp_quota=${inode_quota%%${inode_quota: -1}*}
1626#
1627#    if [ ! ${unit_avail} = ${unit_quota} ] ; then
1628#
1629#    # Convertion
1630#      if [ ${inode_avail: -1} = "T" ] ; then
1631#        (( temp_avail = temp_avail * 1000000000000 ))
1632#      elif [ ${inode_avail: -1} = "G" ] ; then
1633#        (( temp_avail = temp_avail * 1000000000 ))
1634#      elif [ ${inode_avail: -1} = "M" ] ; then
1635#        (( temp_avail = temp_avail * 1000000 ))
1636#      elif [ ${inode_avail: -1} = "k" ] ; then
1637#        (( temp_avail = temp_avail * 1000 ))
1638#      else
1639#        (( temp_avail = inode_avail ))
1640#      fi
1641#
1642#      if [ ${inode_quota: -1} = "T" ] ; then
1643#        (( temp_quota = temp_quota * 1000000000000 ))
1644#      elif [ ${inode_quota: -1} = "G" ] ; then
1645#        (( temp_quota = temp_quota * 1000000000 ))
1646#      elif [ ${inode_quota: -1} = "M" ] ; then
1647#        (( temp_quota = temp_quota * 1000000 ))
1648#      elif [ ${inode_quota: -1} = "k" ] ; then
1649#        (( temp_quota = temp_quota * 1000 ))
1650#      else
1651#        (( temp_quota = inode_quota ))
1652#      fi
1653#    fi
1654#    quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" | bc)
1655##    echo "inode ratio is " $quota_inode
1656#
1657#    if [ ${quota_inode} -ge ${limit_quota} ] ; then
1658#      IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1659#      IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1660#      IGCM_debug_Print 1 "Use the ccc_quota command to check"
1661#      IGCM_debug_Print 1 "You must have more than 10% available to run"
1662#      IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1663#      IGCM_debug_Verif_Exit
1664#    fi
1665#  fi
1666  IGCM_debug_PopStack "IGCM_sys_check_quota"
1667}
1668
1669#D-#==================================================
1670#D-function IGCM_sys_projectAccounting
1671#D-* Purpose: store project accounting information in a file
1672#D-* Examples:
1673#D-
1674function IGCM_sys_projectAccounting {
1675  IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1676  if ( $DEBUG_sys ) ; then
1677    echo "IGCM_sys_check_quota"
1678  fi
1679#need to implemented on Jean Zay
1680  touch $1
1681#  ssh irene191 /usr/bin/ccc_myproject > $1
1682
1683  IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1684}
1685
1686#D-#==================================================
1687#D-function IGCM_sys_getJobSchedulerID
1688#D-* Purpose: Get the job ID during execution
1689#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1690#D-
1691function IGCM_sys_getJobSchedulerID {
1692  IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1693  if ( $DEBUG_sys ) ; then
1694    echo "IGCM_sys_getJobSchedulerID"
1695  fi
1696
1697  eval ${1}=${SLURM_JOBID}
1698
1699  IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1700}
1701
1702#D-#==================================================
1703#D-function IGCM_sys_GetJobID
1704#D-* Purpose: Get the job ID from the JobName
1705#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1706#D-
1707function IGCM_sys_GetJobID {
1708  IGCM_debug_PushStack "IGCM_sys_GetJobID"
1709  if ( $DEBUG_sys ) ; then
1710    echo "IGCM_sys_GetJobID"
1711  fi
1712
1713  ID=$( squeue -u $user | grep Job_${config_UserChoices_JobName} | gawk ' { print $1}' )
1714
1715
1716  eval ${3}=${ID}
1717  IGCM_debug_PopStack "IGCM_sys_GetJobID"
1718}
1719
1720#D-#==================================================
1721#D-function IGCM_sys_CountJobInQueue
1722#D-* Purpose: Count number of users job
1723#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1724#D-
1725function IGCM_sys_CountJobInQueue {
1726  IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1727  if ( $DEBUG_sys ) ; then
1728    echo "IGCM_sys_CountJobInQueue"
1729  fi
1730
1731  # With -f option, the full job name is given in the last column
1732  NbRun=$(squeue -u $user | \
1733      grep -v JOBID | gawk 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1734
1735  eval ${2}=${NbRun}
1736
1737  IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1738}
1739
1740#D-#==================================================
1741#D-function IGCM_sys_ListJobInQueue
1742#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1743#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1744#D-
1745function IGCM_sys_ListJobInQueue {
1746  IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1747  if ( $DEBUG_sys ) ; then
1748    echo "IGCM_sys_ListJobInQueue"
1749  fi
1750
1751# to be implemented on Jean Zay
1752 set -A JobList $( squeue -u $user | gawk '{print $3}' | grep -v NAME)
1753#  # With -f option, the full job name is given in the last column
1754#  set -A JobList $( ccc_mstat -f | gawk -v User=$1             \
1755#                                        '( $2  == User      && \
1756#                                           $NF != /TS/      && \
1757#                                           $NF !~ /PACK/    && \
1758#                                           $NF !~ /REBUILD/ && \
1759#                                           $NF !~ /pack/ )     \
1760#                                         { print $NF }' | sed -e "s/\(.*\)\.[0-9]*/\1/" )
1761#
1762  eval set -A ${2} ${JobList[*]}
1763
1764  IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1765}
1766
1767#D-#==================================================
1768#D-function IGCM_sys_atlas
1769#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1770#D-* Examples:
1771#D-
1772function IGCM_sys_atlas {
1773  IGCM_debug_PushStack "IGCM_sys_atlas" $@
1774  if ( $DEBUG_sys ) ; then
1775    echo "IGCM_sys_atlas :" $@
1776  fi
1777
1778  typeset status
1779# To be implemented on Jean Zay
1780#
1781#  \ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1782#  status=$?
1783#  if [ ${status} -gt 0 ] ; then
1784#    echo "IGCM_sys_atlas : error code ${status}"
1785#    cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1786#    \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1787#    IGCM_debug_PopStack "IGCM_sys_atlas"
1788#    return 1
1789#  else
1790#    IGCM_debug_PopStack "IGCM_sys_atlas"
1791#    return 0
1792#  fi
1793#
1794  IGCM_debug_PopStack "IGCM_sys_atlas"
1795}
1796
1797#D-#==================================================
1798#D-function IGCM_sys_rebuild_nemo
1799#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1800#D-* Examples:
1801#D-
1802
1803function IGCM_sys_rebuild_nemo {
1804  IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1805  if ( $DEBUG_sys ) ; then
1806    echo "IGCM_sys_rebuild_nemo :" $@
1807  fi
1808
1809  /gpfswork/rech/psl/commun/Tools/rebuild_nemo/bin/rebuild_nemo ${1} ${2}
1810
1811  IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1812}
1813
Note: See TracBrowser for help on using the repository browser.