Ignore:
Timestamp:
10/25/23 14:34:02 (7 months ago)
Author:
aclsce
Message:
  • Changed libIGCM behaviour in case of pb in pack_output.job : now, we exit from the job if there is any problem, whatever we are in PROD, DEVT, TEST mode.
  • Specification of pack frequency per file :

In order to reduce the number of inodes, it is possible to specify by file the frequency of packing. The syntax to do that is in the 4th column of OutputFiles? section of the component.card, for example as follows in lmdz.card :
[OutputFiles?]
List= (histmth.nc, ${R_OUT_ATM_O_M}/${PREFIX}_1M_histmth.nc, Post_1M_histmth, 100Y), \

(histday.nc, ${R_OUT_ATM_O_D}/${PREFIX}_1D_histday.nc, Post_1D_histday, 10Y), \

...

In this example, histmth files will be packed every 100 years and histday files will be packed every 10 years.
The pack frequency you defined in config.card is the frequency of pack by default, that means if a specific frequency of pack is specified for a file in a component.card, this file will be packed at the specific frequency whereas all other files will be packed at global pack frequency (specified in config.card) and in this case, the frequency pack (from the config.card) is the frequency the pack_output job will be launched at.
There is a constraint to use this fonctionality : the Packfrequency you defined in config.card must be greater or equal to the pack frequencies you specified for each type of file in component.card, otherwise the computing job will be stopped (with an explicit error message).
Surpack mode :
A surpack mode functionality is available through the use of pack_output.job. To enable this functionality, you have to put "surpack_mode=y" (default value is n). The way to use is similar to restart post-processing pack_output jobs, as indicated here : ​http://forge.ipsl.jussieu.fr/igcmg_doc/wiki/Doc/CheckDebug#RestartPack_output. You can either use a global pack frequency in config.card or specific pack frequency per file, as explained above.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/libIGCM/AA_pack_output

    r1598 r1603  
    155155DEBUG_debug=${DEBUG_debug:=false} 
    156156 
     157#D- Surpack_mode : to pack existing packs output files  
     158#D- Default : n 
     159surpack_mode=n 
     160 
    157161######################################################################## 
    158162 
    … …  
    231235IGCM_debug_Verif_Exit 
    232236 
    233 IGCM_debug_Print 1 "Check coherence between PackFrequency and PeriodLength" 
    234 IGCM_post_CheckModuloFrequency PeriodPack config_UserChoices_PeriodLength NbPeriodPerFrequency 
    235 # ------------------------------------------------------------------ 
    236 # Test if all was right before proceeding further 
    237 # ------------------------------------------------------------------ 
    238 IGCM_debug_Verif_Exit 
    239  
    240 IGCM_debug_Print 1 "We must process ${NbPeriodPerFrequency} files for each pack" 
    241  
    242237# Init loop 
    243238date_begin_pack=${DateBegin} 
    … …  
    248243IGCM_debug_PrintVariables 3 date_end_simulation 
    249244 
    250 while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do 
    251  
    252   IGCM_debug_PrintVariables 3 number_pack 
    253   DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack} ) 
    254   date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) ) 
    255  
    256   for comp in ${config_ListOfComponents[*]} ; do 
     245# Loop over components 
     246for comp in ${config_ListOfComponents[*]} ; do 
     247 
     248    IGCM_card_DefineArrayFromOption ${SUBMIT_DIR}/config.card ListOfComponents ${comp} 
     249    eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1 
     250    card=${SUBMIT_DIR}/COMP/${compname}.card 
     251    #      
     252    if [ X${surpack_mode} = Xy ] ; then 
     253        dirList=$( find ${R_SAVE}/${comp}/Output -maxdepth 1 -mindepth 1 -type d )   
     254    else 
    257255    dirList=$( find ${R_BUFR}/${comp}/Output -maxdepth 1 -mindepth 1 -type d ) 
     256    fi 
     257    # Loop over directories containing output files 
    258258    for dir in ${dirList} ; do 
    259259      # dirID is like ATM.Output.MO 
    … …  
    266266      # Loop over the file type and pack them when in between date_begin_pack and date_end_pack 
    267267      for myType in ${fileType} ; do 
     268                # Read comp.card and find specific pack period for each type of file 
     269                IGCM_card_DefineArrayFromOption ${card} OutputFiles List 
     270                ListFilesName=${compname}_OutputFiles_List 
     271                eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1 
     272 
     273                if [ X${FileName0} != X${NULL_STR} ] ; then 
     274                    eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1 
     275                     
     276                    (( i=0 )) 
     277                    until [ $i -ge $NbFiles ]; do 
     278                        (( i_ = i+1 )) 
     279                        eval file_out_=\${${ListFilesName}[$i_]} > /dev/null 2>&1 
     280                        eval file_out=${file_out_} 
     281                        (( i_ = i+3 )) 
     282                        eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1 
     283                        eval Testvar=${Testvar_} 
     284                        if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then 
     285                            eval PeriodPack_FileType=${Testvar} 
     286                            ((i = i+4)) 
     287                        else 
     288                            PeriodPack_FileType=${PeriodPack} 
     289                            ((i = i+3)) 
     290                        fi 
     291                        if [[ ${file_out} =~ ${myType} ]] ; then 
     292                            break 
     293                        fi 
     294                    done 
     295                fi 
     296                IGCM_debug_Print 1 "Check coherence between PackFrequency and PeriodLength" 
     297                IGCM_post_CheckModuloFrequency PeriodPack_FileType config_UserChoices_PeriodLength NbPeriodPerFrequency 
     298                # ------------------------------------------------------------------ 
     299                # Test if all was right before proceeding further 
     300                # ------------------------------------------------------------------ 
     301                IGCM_debug_Verif_Exit 
     302                # Initialize date_begin_pack 
     303                date_begin_pack=${DateBegin} 
     304                # Loop over period pack 
     305                while [ ${date_begin_pack} -le ${date_end_simulation} ] ; do 
     306                    DaysTemp=$( IGCM_date_DaysInCurrentPeriod ${date_begin_pack} ${PeriodPack_FileType} ) 
     307                    date_end_pack=$( IGCM_date_AddDaysToGregorianDate ${date_begin_pack} $(( ${DaysTemp} - 1 )) )                    
     308                    # 
    268309        grep ${myType} liste_files.${dirID}.txt > liste_files.${dirID}.${myType}.txt 
    269310        nbfile=0 
     311                    # If no surpack, no need to demigrate. 
     312                    if [ X${surpack_mode} = Xn ] ; then 
    270313        for file in $( gawk '{print $11}' liste_files.${dirID}.${myType}.txt ); do 
    271314          extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" ) 
    272315          date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' ) 
    273316          # echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack} 
     317                        # Ajouter un test sur la date 
    274318          if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then 
    275319            echo ${file} >> liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt 
    … …  
    278322          fi 
    279323        done 
     324                    else 
     325                        for file in $( gawk '{print $11}' liste_files.${dirID}.${myType}.txt ); do 
     326                            extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" ) 
     327                            date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' ) 
     328                        # echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack} 
     329                        # Ajouter un test sur la date 
     330                            if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then 
     331                                echo ${file} >> liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt 
     332                                (( nbfile = nbfile + 1 )) 
     333                            fi 
     334                        done 
     335                        # If surpack_mode, archive demigration is needed on TGCC 
     336#-Q- irene-amd ccc_hsm get $( cat liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt ) 
     337#-Q- irene ccc_hsm get $( cat liste_pack_${myType}_${date_begin_pack}_${date_end_pack}.txt ) 
     338                        for file in $( gawk '{print $11}' liste_files.${dirID}.${myType}.txt ); do 
     339                            extract_date_file=$( echo ${file}  | sed -e "s/.*${JobName}_[0-9]*_//" ) 
     340                            date_file=$( echo ${extract_date_file} | sed 's/\([0-9]\{8\}\)_.*$/\1/g' ) 
     341                        # echo pack n°${number_pack}  ${date_file} ${date_begin_pack} ${date_end_pack} 
     342                        # Ajouter un test sur la date 
     343                            if [ ${date_file} -le ${date_end_pack} ] && [ ${date_file} -ge ${date_begin_pack} ] ; then 
     344                                ncdump -h ${file} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> liste_variables_${myType}_${date_begin_pack}_${date_end_pack}.txt 
     345                            fi 
     346                        done 
     347                    fi 
    280348 
    281349        if [ ${nbfile} = 0 ] ; then 
    … …  
    283351          IGCM_debug_Print 1 "We should have found ${NbPeriodPerFrequency} files" 
    284352          IGCM_debug_Print 1 "As some files can be produced only for some selected period we consider we can move to the next file type" 
     353                        date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 ) 
     354                        continue 
     355                    fi 
     356                    if [ X${surpack_mode} = Xy ] &&  [ ${nbfile} = 1 ] ; then 
     357                        IGCM_debug_Print 1 "Only 1 file to pack in surpack mode, nothing to be done" 
     358                        date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 ) 
    285359          continue 
    286360        fi 
    287  
    288361        # Select list of variables to work with 
    289362        list_var=$( cat liste_variables_${myType}_${date_begin_pack}_${date_end_pack}.txt | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 != nbfile) {print $2}}' | paste -s -d ',' ) 
    … …  
    291364        # Create packed files 
    292365        IGCM_debug_Print 1 "Ncrcat ongoing for ${dir} and ${myType}" 
     366                    if [ X${surpack_mode} = Xn ] ; then 
    293367        if [ ! ${nbfile} = ${NbPeriodPerFrequency} ] ; then 
    294368          IGCM_debug_Print 1 "Number of files to process is not equal to what it should be" 
    … …  
    296370          IGCM_debug_Exit "ERROR in number of files to process. STOP HERE INCLUDING THE COMPUTING JOB" 
    297371          IGCM_debug_Verif_Exit 
     372        fi 
    298373        fi 
    299374        output=${JobName}_${date_begin_pack}_${date_end_pack}_${myType} 
    … …  
    309384        IGCM_debug_Verif_Exit 
    310385        # Save it 
     386                    if [ X${surpack_mode} = Xy ] ; then 
     387                        IGCM_sys_Put_Out ${output} ${R_SAVE}/$( echo $dir | sed "s:${R_SAVE}/::" )/${output} 
     388                    else 
    311389        IGCM_sys_Put_Out ${output} ${R_SAVE}/$( echo $dir | sed "s:${R_BUFR}/::" )/${output} 
     390                    fi 
    312391        # Clean file produced by ncrcat 
    313392        IGCM_sys_Rm ${output} 
    … …  
    322401        IGCM_debug_Print 1 "Ncrcat and cleaning done for ${dir} and ${myType}" 
    323402        echo 
     403                    # Add 1 day to date_end_pack to have the new date_begin_pack 
     404                    date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 ) 
    324405      done 
    325406    done 
    326407  done 
    327   (( number_pack = number_pack + 1 )) 
    328   # Add 1 day to date_end_pack to have the new date_begin_pack 
    329   date_begin_pack=$( IGCM_date_AddDaysToGregorianDate ${date_end_pack} 1 ) 
    330408done 
    331409 
    … …  
    336414 
    337415# Flush post-processing submission 
     416if [ X${surpack_mode} = Xn ] && [ ${StandAlone} = false ] ; then 
    338417if [ -f ${R_BUFR}/FlushPost_${DateEnd}.ksh ] ; then 
    339418  . ${R_BUFR}/FlushPost_${DateEnd}.ksh 
    … …  
    341420  #IGCM_sys_Rm -f ${R_BUFR}/FlushPost_${DateEnd}.ksh 
    342421fi 
    343  
     422fi 
    344423# Clean RUN_DIR_PATH (necessary for cesium and titane only) 
    345424IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH} 
Note: See TracChangeset for help on using the changeset viewer.