wiki:Documentation/UserGuide/ReBuild

Where to recover or how to manually rebuild a missing history file

Author: S. Luyssaert
Last revision: 2015/03/23

Note: This page is obsolete. It misses the pack post-proceessing phase which is done before the files are moved to the final archive directory (store).

Identify missing history files

When running a long simulation it sometimes happens that a single year is missing from in your store directory. On curie that would be, for example,

...store/.../IGCM_OUT/OL2/PROD/secsto/JOB_NAME/SBG/Output/MO

Recover the missing files from the scratch

Sometimes libIGCM has difficulties to copy the history files from the scratch to the store. So it is good to first look in the complementary scratch directory.

...scratch.../IGCM_OUT/OL2/PROD/secsto/JOB_NAME/SBG/Output/MO

If the missing output file(s) are there, manually copy them to the store. Remember to copy both the SBG and SRF files.

Rebuild the missing history file(s)

If your run did not crash (check run.card) but there is still a history file missing LibIGCM may have encountered a problem with rebuild job. To check whether this is the case you should look into

...scratch.../IGCM_OUT/OL2/PROD/SECSTO/ACF/REBUILD/REBUILD_XXXX

Where XXXX is the year, month or day for which you are missing an output file. In that folder you should find the same number of stomate and sechiba files as the number of processors you are running with. Thus if you run with 32 procs the rebuild folder should contain 64 files (32 for sechiba and 32 for stomate).

Copy the file rebuild_fromWorkdir.job from the libIGCM folder to your config folder

cp ...work.../DOFOCO.SPINUP/libIGCM/rebuild_fromWorkdir.job /ccc/work/cont003/dsm/cheny/DOFOCO.SPINUP/config/ORCHIDEE_OL/ACF/rebuild_fromWorkdir.job

You will to adjust the job so it does exactly what you want most of the comments are self explaining but NOTE the following: to avoid confusion on the queue (that's how I understood it) LibIGCM starts to rebuild the last year and then goes backward in time. This implies that you have to list the start of the last time step. In the example below the year 1965 will be rebuild. If, for example, you want to rebuild the years 1972 and 1973 then set LastPeriodForRebuild=${LastPeriodForRebuild:=${PeriodDateBegin:=19730101}} and NbRebuildDir=2.

#D- Path to libIGCM
#D- Default : value from AA_job if any
# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
# WARNING you must check MirrorlibIGCM variable in sys library.
# WARNING If this variable is true, you must use libIGCM_POST path instead
# WARNING of your running libIGCM directory.
libIGCM=${libIGCM:=.../DOFOCO.SPINUP/libIGCM}

#-D- $hostname of the MASTER job when SUBMIT_DIR is not visible on postprocessing computer.
MASTER=${MASTER:=ada|curie}

#D- Do we rebuild parallel output from archive or from ${BIGDIR}
#D- Default : value from AA_job if any
RebuildFromArchive=${RebuildFromArchive:=false}

#D- Directory where files we need to rebuild are store
#D- Default : value from AA_job if any
#D- if RebuildFromArchive=true REBUILD_DIR=${DMFDIR}/IGCM_OUT/.../JobName/TMP
#D- example : /dmnfs09/cont003/p86denv/IGCM_OUT/IPSLCM5/CM5PIRC7/TMP
#D- if RebuildFromArchive=false REBUILD_DIR=${BIGDIR}/REBUILD/TagName/JobName/
#D- example : ....scratch.../REBUILD/IPSLCM5/SCAL-NEW
REBUILD_DIR=${REBUILD_DIR:=...scratch.../IGCM_OUT/OL2/PROD/SECSTO/ACF/REBUILD}

#D- How many directory to rebuild we have to consider
#D- Default : value from AA_job if any
NbRebuildDir=1

#D- Suffix date we will use to determine which directory to rebuild
#D- We will rebuild NbRebuildDir before and including PeriodDateBegin
#D- Default : value from AA_job if any
LastPeriodForRebuild=${LastPeriodForRebuild:=${PeriodDateBegin:=19650101}}

Submit the job. If all goes well a new file will be added to the history files. Whether all goes well or something goes wrong a log file will be written in the folder from where you launched the job.

ccc_msub rebuild_fromWorkdir.job

The original instruction can be found at http://forge.ipsl.jussieu.fr/igcmg_doc/wiki/DocGmonitor#Startorrestartpostprocessingjobs1

Rebuild the missing history file(s) - the ultimate solution

On curie you should go to your work directory and try the following command line (adjust the user name and file names)

> .../rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -o ...store.../IGCM_OUT/OL2/PROD/SECSTO/ALFf/SRF/Output/YE/ALFf_19820101_198201231_1Y_sechiba_history.nc ...scratch.../IGCM_OUT/OL2/PROD/SECSTO/ALFf/REBUILD/REBUILD_19820101/sechiba_history_00*

Adjust the statement for the stomate_history files.

Last modified 10 months ago Last modified on 04/01/20 17:41:17