${RUN_DIR} filesystem failure before BigBrother has started resulting as an uncatch error ...

[sdipsl]

mkdir -p /ccc/scratch/cont003/gencmip6
mkdir: cannot create directory `/ccc/scratch/cont003/gencmip6'

Workaround : if a filesystem is messy like below then try to find a backup plan to stop smoothly, not like below.

/ccc/cont003/home/gencmip6/p86denv/IPSLCM6.0.5-LR/config/IPSLCM6/CM605.WF01.COMPRESSeq6/Script_Output_CM605.WF01.COMPRESSeq6.000076.SAVE

+ SCRIPT_PID=92254
+ set +x
+ /bin/ksh -x /tmp/jobstart.92240
+ BATCH_NUM_PROC_TOT=557
Sat Sep 10 17:17:43 CEST 2016

#######################################
# ANOTHER GREAT SIMULATION #
#######################################

====================================================
Where do we run ? curie1554

2016-09-10 17:17:45 --Debug1--> R_OUT has been defined = /ccc/store/cont003/gencmip6/p86denv/IGCM_OUT
2016-09-10 17:17:45 --Debug1--> R_BUF has been defined = /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT
2016-09-10 17:17:45 --Debug1--> R_FIG has been defined = /ccc/work/cont003/gencmip6/p86denv/IGCM_OUT
IGCM_sys_MkdirWork : /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/REBUILD
mkdir: cannot create directory `/ccc/scratch/cont003/gencmip6': File exists
IGCM_sys_MkdirWork : erreur.
IGCM_debug_Exit :  IGCM_sys_MkdirWork

!!!!!!!!!!!!!!!!!!!!!!!!!!
!!   ERROR TRIGGERED    !!
!!   EXIT FLAG SET      !!
!------------------------!

0 - IGCM_config_CommonConfiguration (/ccc/cont003/home/gencmip6/p86denv/IPSLCM6.0.5-LR/config/IPSLCM6/CM605.WF01.COMPRESSeq6/config.card)
 1 - IGCM_sys_MkdirWork (/ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/REBUILD)
  2 - IGCM_debug_Exit (_0_)
mkdir: cannot create directory `/ccc/scratch/cont003/gencmip6': File exists
/tmp/jobstart.92240[263]: IGCM_config_CommonConfiguration[187]: IGCM_debug_PopStack[599]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
2016-09-10 17:17:45 --------Debug2--> run.card exists
/tmp/jobstart.92240[281]: IGCM_debug_BigBro_Initialize[648]: IGCM_debug_PushStack[372]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
/tmp/jobstart.92240[281]: IGCM_debug_BigBro_Initialize[648]: IGCM_debug_PushStack[380]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
/tmp/jobstart.92240[281]: IGCM_debug_BigBro_Initialize[663]: IGCM_sys_getJobSchedulerID[1589]: IGCM_debug_PushStack[372]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
/tmp/jobstart.92240[281]: IGCM_debug_BigBro_Initialize[663]: IGCM_sys_getJobSchedulerID[1589]: IGCM_debug_PushStack[375]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
/tmp/jobstart.92240[281]: IGCM_debug_BigBro_Initialize[663]: IGCM_sys_getJobSchedulerID[1589]: IGCM_debug_PushStack[380]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq6/Out/computing.stack.92254: cannot create [No such file or directory]
IGCM_sys_getJobSchedulerID
[...]

[sdipsl]

another example.

+ SCRIPT_PID=17564
+ set +x
+ /bin/ksh -x /tmp/jobstart.17550
+ BATCH_NUM_PROC_TOT=557
Mon Sep 12 10:44:52 CEST 2016

#######################################
# ANOTHER GREAT SIMULATION #
#######################################

====================================================
Where do we run ? curie2065

2016-09-12 10:44:54 --Debug1--> IGCM_config_CommonConfiguration
2016-09-12 10:44:54 --Debug1--> /ccc/cont003/home/gencmip6/p86denv/IPSLCM6.0.5-LR/config/IPSLCM6/CM605.WF01.COMPRESSeq9/config.card

2016-09-12 10:44:54 --Debug1--> DefineArrayFromOption : config_UserChoices
2016-09-12 10:44:54 --------------Debug3--> JobName : CM605.WF01.COMPRESSeq9
2016-09-12 10:44:54 --------------Debug3--> ExperimentName : pdControl
2016-09-12 10:44:54 --------------Debug3--> SpaceName : DEVT
2016-09-12 10:44:54 --------------Debug3--> LongName : IPSLCM6.0.5 test with new workflow and compression
2016-09-12 10:44:54 --------------Debug3--> TagName : IPSLCM6
2016-09-12 10:44:54 --------------Debug3--> ExpType : IPSLCM/pdControl
2016-09-12 10:44:54 --------------Debug3--> CalendarType : noleap
2016-09-12 10:44:54 --------------Debug3--> DateBegin : 1950-01-01
2016-09-12 10:44:54 --------------Debug3--> DateEnd : 2049-12-31
2016-09-12 10:44:54 --------------Debug3--> ORCA_version : eORCA1.2
2016-09-12 10:44:54 --------------Debug3--> PeriodLength : 1Y

2016-09-12 10:44:55 --Debug1--> DefineArrayFromOption : config_Ensemble

2016-09-12 10:44:55 --Debug1--> Warning with readding of config_Ensemble variable in /ccc/cont003/home/gencmip6/p86denv/IPSLCM6.0.5-LR/config/IPSLCM6/CM605.WF01.COMPRESSeq9/config.card.
2016-09-12 10:44:55 --Debug1--> Warning: Section Ensemble not found

2016-09-12 10:44:55 --Debug1--> DefineArrayFromOption : config_Post
2016-09-12 10:44:55 --------------Debug3--> RebuildFrequency : NONE
2016-09-12 10:44:55 --------------Debug3--> PackFrequency : 10Y
2016-09-12 10:44:55 --------------Debug3--> SeasonalFrequency : 10Y
2016-09-12 10:44:55 --------------Debug3--> SeasonalFrequencyOffset : 0

2016-09-12 10:44:55 --Debug1--> R_OUT has been defined = /ccc/store/cont003/gencmip6/p86denv/IGCM_OUT
2016-09-12 10:44:55 --Debug1--> R_BUF has been defined = /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT
2016-09-12 10:44:55 --Debug1--> R_FIG has been defined = /ccc/work/cont003/gencmip6/p86denv/IGCM_OUT
IGCM_sys_MkdirWork : /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/REBUILD
mkdir: cannot create directory `/ccc/scratch/cont003/gencmip6': File exists
IGCM_sys_MkdirWork : erreur.
IGCM_debug_Exit :  IGCM_sys_MkdirWork

!!!!!!!!!!!!!!!!!!!!!!!!!!
!!   ERROR TRIGGERED    !!
!!   EXIT FLAG SET      !!
!------------------------!

0 - IGCM_config_CommonConfiguration (/ccc/cont003/home/gencmip6/p86denv/IPSLCM6.0.5-LR/config/IPSLCM6/CM605.WF01.COMPRESSeq9/config.card)
 1 - IGCM_sys_MkdirWork (/ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/REBUILD)
  2 - IGCM_debug_Exit (_0_)
mkdir: cannot create directory `/ccc/scratch/cont003/gencmip6': File exists
/tmp/jobstart.17550[263]: IGCM_config_CommonConfiguration[187]: IGCM_debug_PopStack[599]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/Out/computing.stack.17564: cannot create [No such file or directory]
2016-09-12 10:44:55 --------Debug2--> run.card exists
/tmp/jobstart.17550[281]: IGCM_debug_BigBro_Initialize[648]: IGCM_debug_PushStack[372]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/Out/computing.stack.17564: cannot create [No such file or directory]
/tmp/jobstart.17550[281]: IGCM_debug_BigBro_Initialize[648]: IGCM_debug_PushStack[380]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/Out/computing.stack.17564: cannot create [No such file or directory]
/tmp/jobstart.17550[281]: IGCM_debug_BigBro_Initialize[663]: IGCM_sys_getJobSchedulerID[1589]: IGCM_debug_PushStack[372]: /ccc/scratch/cont003/gencmip6/p86denv/IGCM_OUT/IPSLCM6/DEVT/pdControl/CM605.WF01.COMPRESSeq9/Out/computing.stack.17564: cannot create [No such file or directory]

[sdipsl]

Never happened again. keep things simple.