Changeset 201 for trunk


Ignore:
Timestamp:
07/26/13 16:21:43 (11 years ago)
Author:
pinsard
Message:

add some update data tools

Location:
trunk
Files:
1 added
1 edited
3 copied

Legend:

Unmodified
Added
Removed
  • trunk/docs/docs_dev/source/guides/updatedata.rst

    r196 r201  
    99.. ======== 
    1010.. 
    11 .. ref:`updatedata` 
    1211.. ref:`builddoc` 
    1312.. 
     
    3736.. 
    3837.. $URL$ 
     38.. 
     39.. - fplod 20130725T141747Z cratos.locean-ipsl.upmc.fr (Linux) 
     40.. 
     41..   * detailed specific task for 20130715 update 
    3942.. 
    4043.. - fplod 20121227T192052Z cratos.locean-ipsl.upmc.fr (Linux) 
     
    8790 - mettre ce nouveau jeu de données à disposition sous http://www.locean-ipsl.upmc.fr/~tropflux/data/ 
    8891 - prévenir les amis de TropFlux (cf. :ref:`advertise`) 
     92 
     93.. note:: 
     94 
     95   see detailed and specific tasks for each update below. 
    8996 
    9097.. _newversion: 
     
    229236 
    230237      $ diskext="/Volumes/LCerbere" 
     238      $ mkdir ${diskext} 
    231239      $ sshfs fplod@cerbere:/usr/lodyc/incas/fplod ${diskext} -odefer_permissions -o sshfs_debug 
    232240 
     
    240248Une mise à jour doit être envoyée aux contacts dans les instutions qui mentionne notre jeu de données (NIO). 
    241249 
     250 
     251.. toctree:: 
     252   :maxdepth: 0 
     253   :glob: 
     254 
     255   updatedata/updatedata_20130725 
  • trunk/docs/docs_dev/source/guides/updatedata/updatedata_20130725.rst

    r199 r201  
    44.. =========== 
    55.. 
    6 .. procedure to update data on website 
     6.. procedure to update data on website 20130725 
    77.. 
    88.. SEE ALSO 
     
    1010.. 
    1111.. ref:`updatedata` 
    12 .. ref:`builddoc` 
    1312.. 
    1413.. TODO 
    1514.. ===== 
    1615.. 
    17 .. en fait la comparaison devra être fait aussi AVANT le démarage de cette procédure 
    18 .. 
    19 .. translation 
    20 .. 
    21 .. comprendre les embrouilles ftp, sftp ncftp www.locean-ipsl.upmc.fr 
    22 .. 
    23 .. mettre les sites amis comme le NIO à jour 
    24 .. 
    25 .. définir les outils de comparaison, écrire les outils en question 
    26 .. 
    27 .. tenir compte des images du site dans cette procédure 
    28 .. 
    29 .. tester la mise à jour de Readme.rst 
    30 .. 
    31 .. utilisation du .htaccess 
    32 .. 
    3316.. EVOLUTIONS 
    3417.. ========== 
     
    3821.. $URL$ 
    3922.. 
    40 .. - fplod 20121227T192052Z cratos.locean-ipsl.upmc.fr (Linux) 
    41 .. 
    42 ..   * no more need to concatenate /net/adonis/usr/adonis/varclim/tropflux_v2 
    43 ..     : jv made it on 20121221 
    44 .. 
    45 .. - fplod 20121121T151409Z cratos.locean-ipsl.upmc.fr (Linux) 
    46 .. 
    47 ..   * update from /net/adonis/usr/adonis/varclim/tropflux_v2 
    48 ..     2 datasets : 1979-2011 and 201201-201206 
    49 ..     to be concatenated 
    50 .. 
    51 .. - fplod 20120307 
    52 .. 
    53 ..   * README.rst was renamed to Readme.rst because of Apache policy 
    54 .. 
    55 .. - fplod 20120229 
    56 .. 
    57 ..   * associate README.rst to datafiles 
    58 .. 
    59 .. - fplod 20120210 
    60 .. 
    61 ..   * missing an ftp command 
    62 .. 
    63 .. - fplod 20120209 
    64 .. 
    65 ..   * creation (draft) with v0 to v1 action 
     23.. - fplod 20130725T141747Z cratos.locean-ipsl.upmc.fr (Linux) 
     24.. 
     25..   * creation 
    6626.. 
    6727.. - 
     
    7030.. |project_id| replace:: tropflux 
    7131 
    72 .. _updatedata: 
    73  
    74 Update data 
    75 =========== 
    76  
    77 On suppose ici que : 
    78  
    79  - un nouveau jeu de données est disponible dans un répertoire accessible à partir de cratos.locean-ipsl.upmc.fr (pour faire tourner les outils de validation/comparaison) et à partir du poste de travail (mac) de Françoise Pinsard (pour assurer la mise à jour du serveur de données). 
    80  - (bientôt) des nouvelles images associées sont disponibles dans les mêmes conditions 
    81  
    82 Le but du jeu de la procédure décrite ici est de : 
    83  
    84  - conserver le jeu de données actuel (cf. :ref:`saveold`) 
    85  - conserver le jeu de données nouveau (cf. :ref:`copynew1` and :ref:`copynew2`) 
    86  - comparer les deux jeux de données (cf. :ref:`compareversion`) 
    87  - mettre ce nouveau jeu de données à disposition sous http://www.locean-ipsl.upmc.fr/~tropflux/data/ 
    88  - prévenir les amis de TropFlux (cf. :ref:`advertise`) 
    89  
    90 .. _newversion: 
    91  
    92 It must be known if the new dataset is a new version of TropFlux product. 
    93  
    94 If so, the new version must be described in :file:`changes.rst`. 
    95 (see also :ref:`builddoc` to synchronize dataset and documentation). 
    96  
    97 If a new field is provided it must be described in :file:`data_out_website.rst` 
    98  
    99 :file:`Readme.rst` may also be updated. 
    100  
    101 .. warning:: 
    102  
    103    embrouille entre les machines (cratos vs mon poste de travail) et les outils (ftp, ncftp, sftp) : on arrive pas toujours à trouver le répertoire  |project_id| après la connexion. 
    104  
    105    En conséquence, toutes les commandes ftp sont faites sur le poste de travail (mac) de Françoise Pinsard. 
    106  
    107 .. note:: 
    108  
    109    Here is the sequence of commands to be run to create 
    110    the directory equivalent to the URL  http://www.locean-ipsl.upmc.fr/~tropflux/images/ 
    111  
    112    .. parsed-literal:: 
    113  
    114       fplod@sonmac$ ftp www.locean-ipsl.upmc.fr 
    115       ftp> cd |project_id| 
    116       ftp> mkdir images 
    117  
    118 La mise à disposition des fichiers de données et des images sur l'URL 
    119 http://www.locean-ipsl.upmc.fr/tropflux/ 
    120 est réservée à Françoise Pinsard pour des raisons de protections de fichiers et 
    121 de politique de login. 
     32.. _updatedata_20130725: 
     33 
     34Update data 20130725 
     35==================== 
     36 
     37Here are details tasks for the update data 20130725 derived from :ref:`updatedata`. 
     38 
     39.. _newversion_20130725: 
     40 
     41Only time coverage extension from July 2012 to March 2013 
     42 
     43:file:`${PROJECT}/website/data/changes.rst` and :file:`${PROJECT}/adm/website/Readme.rst` are updated.  
    12244 
    12345.. _saveold: 
     
    12850.. parsed-literal:: 
    12951 
    130    $ datemodif="20120209" 
     52   $ datemodif="20130725" 
    13153 
    13254   $ export datemodif 
     
    14062   $ mv ./monthly obsolete_${datemodif} 
    14163 
    142 .. _copynew1: 
     64.. _copynew1_20130725: 
    14365 
    14466Copy new version (phase 1) 
     
    14769.. parsed-literal:: 
    14870 
    149    $ newdir='/net/adonis/usr/adonis/varclim/tropflux_v1' 
     71   $ mkdir /usr/lodyc/incas/fplod/tropflux_d/to_be_published 
     72   $ newdir='/net/adonis/usr/adonis/varclim/tropflux_v2' 
    15073   $ export newdir 
    151    $ svn export svn+ssh://pinsard@forge.ipsl.jussieu.fr/ipsl/forge/projets/tropflux/svn/trunk/adm/website/Readme.rst \ 
    152      /usr/lodyc/incas/fplod/tropflux_d/to_be_published/Readme.rst 
     74 
     75Is there enought space ? 
     76   $ du -sh ${newdir}/daily 
     77   12G     /net/adonis/usr/adonis/varclim/tropflux_v2/daily 
     78   $ du -sh ${newdir}/monthly 
     79   396M    /net/adonis/usr/adonis/varclim/tropflux_v2/monthly 
     80   $ df -h /usr/lodyc/incas/fplod/tropflux_d/to_be_published 
     81   Filesystem            Size  Used Avail Use% Mounted on 
     82   fslodyc:/usr/.lodyc   1.6T  1.6T   20G  99% /.autofs/lodyc 
     83 
     8420GB vs 12.4 GB = yes 
     85 
     86   $ cp ${PROJECT}/adm/website/Readme.rst /usr/lodyc/incas/fplod/tropflux_d/to_be_published/Readme.rst 
    15387   $ cp -r ${newdir}/daily /usr/lodyc/incas/fplod/tropflux_d/to_be_published 
    15488   $ cp -r ${newdir}/monthly /usr/lodyc/incas/fplod/tropflux_d/to_be_published 
    15589 
    156 .. _compareversion: 
     90.. _compareversion_20130725: 
    15791 
    15892Compare versions 
     
    167101   $ ++compare.sh -o obsolete_${datemodif} -n /usr/lodyc/incas/fplod/tropflux_d/to_be_published 
    168102 
     103   mkdir -p /usr/work/incas/fplod/tropflux_d//compare_update/ 
     104   varlist="lhf lwr netflux q2m shf swr t2m tau taux tauy ws" 
     105   period_list="daily monthly" 
     106   lastyear=2012 
     107   for var in $varlist 
     108   do 
     109      for period in ${period_list} 
     110      do  
     111         if [ "${period}" == "daily" ] 
     112         then  
     113            suffix=1d 
     114         else 
     115            suffix=1m 
     116         fi 
     117         newfile=/usr/lodyc/incas/fplod/tropflux_d/to_be_published/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 
     118         newfile_oldtimecoverage=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastdate}.nc 
     119         # extract old time coverage in the new file 
     120         nt=$(ncdump -h $oldfile | grep "time = UNLIMITED ;" | sed -e "s@\(time = UNLIMITED ; // (\)\(.*\)\( currently)\)@\2@") 
     121         slabtime=$(printf "time,0,%d" $((${nt} -1 ))) 
     122         ncks -d ${slabtime} -O -o ${newfile_oldtimecoverage} ${newfile} 
     123         oldfile=/usr/lodyc/incas/fplod/tropflux_d/obsolete_${datemodif}/${period}/${var}_tropflux_${suffix}_1979_${lastyear}.nc 
     124         oldfiledump=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastyear}.nc_ncdump 
     125         cdo diff  ${newfile_oldtimecoverage} ${oldfile} 
     126         read a 
     127         # we must see "0 of $nt records differ" 
     128         ncdump -h ${newfile_oldtimecoverage} > ${newfile_oldtimecoverage}_ncdump 
     129         ncdump -h ${oldfile} > ${oldfiledump} 
     130         diff ${newfile_oldtimecoverage}_ncdump ${oldfiledump} 
     131         # we must see a difference in creation date and coverage 
     132         read a 
     133         rm ${newfile_oldtimecoverage}_ncdump 
     134         rm ${oldfiledump} 
     135         unset suffix 
     136      done 
     137      unset period 
     138      unset period_list 
     139   done 
     140   unset var 
     141   unset varlist 
     142   rm -rf /usr/work/incas/fplod/tropflux_d//compare_update/ 
     143 
     144   log=${PROJECT_LOG}/compare.log 
     145   script ${PROJECT_LOG}/compare.log 
     146   compare.sh 
     147   exit 
     148 
     149Check for results. 
     150 
     151First check expected equality for data: 
     152 
     153:: 
     154 
     155   grep "records differ" ${log} 
     156   
     157If only "0 of" begin all lines, its ok. Otherwise look inside the log file to 
     158locate troubles. 
     159 
     160Second, check for differences on global attributes 
     161 
     162:: 
     163 
     164   grep "^>" ${log} | grep creation_date 
     165 
     166If nothing appears, you must modified this attributes. see :ref:`change_creation_date.sh` ++ 
     167 
     168 
     169   grep "^>" ${log} | grep time_range 
     170 
     171If nothing appears, you must modified this attributes. see :ref:`change_timerange.sh` ++ 
     172 
     173Clean log file:: 
     174 
     175   rm ${log} 
     176 
     177If modifications has been made, re-run comparaison before next step 
     178 
    169179Once comparison is done and PI happy, 
    170180++ cleanning or modify newdir 
    171181 
    172 .. _copynew2: 
     182.. _copynew2_20130725: 
    173183 
    174184Copy new version (phase 2) 
     
    185195 
    186196 
    187 .. _desinstalldata: 
     197.. _desinstalldata_20130725: 
    188198 
    189199Remove old version from the website 
     
    199209   ftp> mdelete monthly/* 
    200210 
    201 .. _installdata: 
     211.. _installdata_20130725: 
    202212 
    203213Installation of the new version 
     
    229239 
    230240      $ diskext="/Volumes/LCerbere" 
     241      $ mkdir ${diskext} 
    231242      $ sshfs fplod@cerbere:/usr/lodyc/incas/fplod ${diskext} -odefer_permissions -o sshfs_debug 
    232243 
    233 .. _advertise: 
    234  
    235 Advertise TropFlux friends 
    236 -------------------------- 
    237  
    238 A email must be send to TropFlux list to warn about the new dataset. 
    239  
    240 Une mise à jour doit être envoyée aux contacts dans les instutions qui mentionne notre jeu de données (NIO). 
    241  
     244 
     245 et template pour NEW 
     246 
     247et build doc et install doc 
     248 
     249(see also :ref:`builddoc` to synchronize dataset and documentation). 
     250 
     251 
     252et svn commit ppur ce fichier et tous ceux mentionnes ici 
     253rm /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ 
     254 
  • trunk/src/change_creation_date.sh

    r199 r201  
    33#+ 
    44# 
    5 # .. program:: add_19890101.sh 
    6 # 
    7 # .. _add_19890101.sh: 
    8 # 
    9 # ================== 
    10 # add_19890101.sh  
    11 # ================== 
     5# .. program:: change_creation_date.sh 
     6# 
     7# .. _change_creation_date.sh: 
     8# 
     9# ======================= 
     10# change_creation_date.sh 
     11# ======================= 
    1212# 
    1313# SYNOPSIS 
     
    1616# :: 
    1717# 
    18 #  $ add_19890101.sh [--debug] 
     18#  $ change_creation_date.sh [--debug] -c yyyymmdd -d directory 
    1919# 
    2020# DESCRIPTION 
     
    2525#    If this option is set, :samp:`ncdump -v tt` will be added to log file 
    2626# 
    27 # Artificialy add 19890101 to lwr interpolated on OAFLUX grid dataset. 
    28 # 
    29 # :file:`${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc` 
    30 # containing  
    31 # ++ 
    32 # has been produced by 
    33 # :func:`interp_erai_lwr`. 
    34 # repeat the first time step in  
    35 # 
    36 # File :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 
    37 # will be written 
    38 # if this file not already exists. 
    39 # 
    40 # This output file  
    41 # :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 
    42 # must be processed after by  
    43 # :func:`lwr_correction_ncdf`. 
    44 #  
    45 # Log file is written on :file:`${PROJECT_LOG}/add_19890101.sh.log.{YYYYMMDDTHHMMSSZ}` 
     27# .. option:: -c <yyyymmdd> 
     28# 
     29#    date to be written in the global attributes creation_date 
     30# 
     31# .. option:: -d <dirin> 
     32# 
     33#    base location of the dataset to be changed 
     34# 
     35# Modify creation_date global attribute if the dataset based under dirin parameter 
     36# 
     37# Log file is written on :file:`${PROJECT_LOG}/change_creation_date.sh.log.{YYYYMMDDTHHMMSSZ}` 
    4638# 
    4739#     .. graphviz:: 
    4840# 
    49 #        digraph add_19890101 { 
    50 # 
    51 #           filein [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc"]; 
    52 #           fileout [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc"]; 
    53 # 
    54 #           add_19890101 [shape=box, 
     41#        digraph change_creation_date { 
     42# 
     43#           change_creation_date [shape=box, 
    5544#           fontname=Courier, 
    5645#           color=blue, 
    57 #           URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/add_19890101.sh", 
    58 #           label="${PROJECT}/src/add_19890101.sh"]; 
    59 # 
    60 #           {filein} -> {add_19890101} -> {fileout} 
     46#           URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/change_creation_date.sh", 
     47#           label="${PROJECT}/src/change_creation_date.sh"]; 
     48# 
     49#           {filein} -> {change_creation_date} -> {filein} 
    6150# 
    6251#       } 
     
    6554# ======== 
    6655# 
    67 # To produce $PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc  
    68 # from $PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc :: 
    69 # 
    70 #  $ add_19890101.sh --debug 
     56# To modify files under /usr/lodyc/incas/fplod/tropflux_d/to_be_published/:: 
     57# 
     58#  $ change_creation_date.sh -d /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ -c 20130725 
    7159# 
    7260# And look at log file with :: 
    7361# 
    74 #  $ tlogd.sh add_19890101 
    75 # 
    76 # and of course on files in ${PROJECT_OD}. 
     62#  $ tlogd.sh change_creation_date 
    7763# 
    7864# TIPS 
    7965# ==== 
    8066# 
    81 # To see the 3 first time values in input and files:: 
    82 # 
    83 #  $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc  
    84 #    14246.0000000005 
    85 #    14247.0000000005 
    86 #    14248.0000000005 
    87 # 
    88 #  $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc  
    89 #    14245.0000000005 
    90 #    14246.0000000005 
    91 #    14247.0000000005 
    92 # 
    93 # while : 
    94 # 
    95 #   $ idl 
    96 #   print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) 
    97 #   14245.500 
    98 #   print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) +1 
    99 #   14246.500 
    100 # 
    10167# SEE ALSO 
    10268# ======== 
    10369# 
     70# :ref:`updatedata` 
     71# 
    10472# :ref:`project_profile.sh` 
    10573# 
    106 # :func:`ncks <nco:ncks>` 
    107 # :func:`ncrcat <nco:ncrcat>` 
    10874# :func:`ncatted <nco:ncatted>` 
    109 # :func:`ncap2 <nco:ncap2>` 
    110 # 
    111 # :ref:`lwr_correction_ncdf.pro` 
    11275# 
    11376# TODO 
     
    11679# coding rules 
    11780# 
    118 # won't it be more conveniant to add this articifial time step in  
    119 # the IDL creation tool interp_erai_lwr_1989_2009.pro. 
    120 # It will allow to avoid a shell interruption in the IDL processing chain. 
    121 # 
    122 # improve global attributes with ... creation_date and so on 
    123 # 
    124 # check if the first time step is really 19890102 in the input file 
    125 # 
    126 # change tt to ttt to complete time correction 
    127 # 
    128 # remove hard coded end time 
    129 # 
    130 # get rid of this long mesage on cratos:: 
    131 # 
    132 #    ncrcat: WARNING Variable lwr has attribute "missing_value" but not "_FillValue". To  
    133 #    comply with netCDF conventions, NCO ignores values that equal the _FillValue attribu 
    134 #    te when performing arithmetic. Confusingly, values equal to the missing_value should 
    135 #     also be neglected. However, it is tedious and (possibly) computationally expensive  
    136 #    to check each value against multiple missing values during arithmetic on large varia 
    137 #    bles. So NCO thinks that processing variables with a "missing_value" attribute and n 
    138 #    o "_FillValue" attribute may produce undesired arithmetic results (i.e., where value 
    139 #    s that were intended to be neglected were not, in fact, neglected). We suggest you r 
    140 #    ename all "missing_value" attributes to "_FillValue" or include both "missing_value" 
    141 #     and "_FillValue" attributes (with the _same values_) for all variables that have ei 
    142 #    ther attribute. Because it is long, this message is only printed once per operator e 
    143 #    ven though multiple variables may have the same attribute configuration. More inform 
    144 #    ation on missing values is given at: 
    145 #    http://nco.sf.net/nco.html#mss_val 
    146 #    Examples of renaming attributes are at: 
    147 #    http://nco.sf.net/nco.html#xmp_ncrename 
    148 # 
    149 # ... on a déjà une idee de la solution : convention CF et meilleure gestion des 
    150 # valeurs manquantes dans les softs IDL !!! 
    151 # 
     81# overwrite : may be an other option to avoid scratch 
     82# 
     83# debug not used 
     84# 
     85# terminology march2013 non normative 
    15286# 
    15387# EVOLUTIONS 
     
    15892# $URL$ 
    15993# 
    160 # - fplod 20110819T091143Z aedon.locean-ipsl.upmc.fr (Darwin) 
    161 # 
    162 #   * creation from pk email 20110811 "lwr first timestep" 
    163 #   * include time correction and timerange modificaton 
     94# - fplod 20130726T112836Z cratos.locean-ipsl.upmc.fr (Linux) 
     95# 
     96#   * creation 
    16497# 
    16598#- 
     
    191124log_date=$(date -u +"%Y%m%dT%H%M%SZ") 
    192125# 
    193 usage=" Usage : ${command} [--debug]" 
     126usage=" Usage : ${command} [--debug] -d dirin -c yyyymmdd" 
    194127# 
    195128hostname=$(hostname) 
     
    209142do 
    210143   case ${1} in 
     144      -c) 
     145         yyyymmdd=${2} 
     146         shift 
     147      ;; 
     148      -d) 
     149         dirin=${2} 
     150         shift 
     151      ;; 
    211152      --debug) 
    212153         debug=1 
     
    224165# 
    225166# check parameters 
    226 # N.A. 
    227 # 
    228 tool=ncrcat 
    229 type ${tool} 1> /dev/null 2>&1 
    230 status=${?} 
    231 if [ ${status} -ne 0 ] 
    232 then 
    233    echo "${command} : eee : tool ${tool} not found" 
    234    exit 1 
    235 fi 
    236 unset status 
    237 unset tool 
    238 # 
    239 tool=ncks 
    240 type ${tool} 1> /dev/null 2>&1 
    241 status=${?} 
    242 if [ ${status} -ne 0 ] 
    243 then 
    244    echo "${command} : eee : tool ${tool} not found" 
    245    exit 1 
    246 fi 
    247 unset status 
    248 unset tool 
     167if [ ! -d ${dirin} ] 
     168then 
     169   echo " eee : ${dirin} not found" 
     170   exit 1 
     171fi 
     172# 
     173# check for permission on dirin 
     174if [ ! -x ${dirin} ] 
     175then 
     176   echo " eee : ${dirin} not reachable" 
     177   exit 1 
     178fi 
     179#++ check yyyymmdd validity 
    249180# 
    250181tool=ncatted 
     
    253184if [ ${status} -ne 0 ] 
    254185then 
    255    echo "${command} : eee : tool ${tool} not found" 
     186   echo "${command} : ${LINENO} : eee : tool ${tool} not found" 
    256187   exit 1 
    257188fi 
     
    262193if [ "${PROJECT_LOG}" = "" ] 
    263194then 
    264    echo "${command} : eee : \${PROJECT_LOG} not defined" 
     195   echo "${command} : ${LINENO} : eee : \${PROJECT_LOG} not defined" 
    265196   exit 1 
    266197fi 
     
    297228echo "" 1>> ${log} 
    298229echo "[Parameters]" 1>> ${log} 
    299 echo "N.A." 1>> ${log} 
     230echo "dirin=${dirin}" 1>> ${log} 
     231echo "yyyymmdd=${yyyymmdd}" 1>> ${log} 
    300232echo "" 1>> ${log} 
    301233# 
    302 filein=${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc 
    303 if [ ! -f ${filein} ] 
    304 then 
    305     echo " eee : ${filein} not found" 1>> ${log} 
    306     exit 1  
    307 fi 
    308 fileout=${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc 
    309 if [ -f ${fileout} ] 
    310 then 
    311       echo "eee : ${fileout} already exists" 1>>${log} 
    312       exit 1 
    313 fi 
    314 # 
    315 rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 
    316 rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 
    317 # 
    318 # extraction of the first time step 19890102 
    319 ncks -h -d time,0,0 ${filein} ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    320 status=${?} 
    321 if [ ${status} -ne 0 ] 
    322 then 
    323     echo "eee : pb with ncks" >> ${log} 2>&1 
    324     ncdump -h ${filein} >> ${log} 2>&1 
    325     exit 1 
    326 fi 
    327 unset status 
    328 # 
    329 # ++ check if tt[0] is really 19890102 
    330 # 
    331 # substract 1 to tt to simulate 19890101 
    332 ncap2 -h -s "tt=tt-1" \ 
    333     ${PROJECT_OD}/erai_lwr_step19890102.nc \ 
    334     ${PROJECT_OD}/erai_lwr_step19890101.nc 
    335 status=${?} 
    336 if [ ${status} -ne 0 ]  
    337 then 
    338     echo "eee : pb with ncap2" >> ${log} 2>&1 
    339     ncdump -h ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    340     exit 1 
    341 fi   
    342 unset status 
    343 ls -l ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    344 ls -l ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 
    345 # 
    346 # ++ check if tt[0] is really 19890101 
    347 #  
    348 rm ${fileout}_bad_timerange 2> /dev/null 
    349 ncrcat -h \ 
    350        ${PROJECT_OD}/erai_lwr_step19890101.nc \ 
    351        ${filein} \ 
    352        ${fileout}_bad_timerange >> ${log} 2>&1 
    353 status=${?} 
    354 if [ ${status} -ne 0 ]  
    355 then 
    356     echo "eee : pb with ncrcat" >> ${log} 2>&1 
    357     ncdump -h ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 
    358     ncdump -h ${filein} >> ${log} 2>&1 
    359     exit 1 
    360 fi   
    361 unset status 
    362 # 
    363 ncatted -h -a timerange,global,o,c,"19890101 - 20091231" \ 
    364        ${fileout}_bad_timerange \ 
    365        ${fileout} >> ${log} 2>&1 
    366 status=${?} 
    367 if [ ${status} -ne 0 ] 
    368 then 
    369     echo "eee : pb with ncatted" >> ${log} 2>&1 
    370     ncdump -h ${fileout}_bad_timerange >> ${log} 2>&1 
    371     exit 1 
    372 fi  
    373 unset status 
    374 # 
    375 if [ ${debug} -eq 1 ] 
    376 then 
    377     ncdump -v tt ${fileout} >> ${log} 2>&1 
    378 fi 
     234# build list files 
     235varlist="lhf lwr netflux q2m shf sst swr t2m tau taux tauy ws" 
     236period_list="daily monthly" 
     237list_filein="" 
     238for var in $varlist 
     239do 
     240     for period in ${period_list} 
     241     do 
     242         if [ "${period}" == "daily" ] 
     243         then 
     244            suffix=1d 
     245         else 
     246            suffix=1m 
     247         fi 
     248         filein=${dirin}/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 
     249         if [ ! -f ${filein} ] 
     250         then 
     251             echo " eee : ${filein} not found" 1>> ${log} 
     252             exit 1 
     253         else 
     254             list_filein=${list_filein}" ${filein}" 
     255         fi 
     256     done 
     257     unset period 
     258done 
     259unset period_list 
     260unset var 
     261unset varlist 
     262# 
     263# new creation_date attribute 
     264for filein in ${list_filein} 
     265do 
     266    echo "iii : change creation_date in ${filein}" >> ${log} 2>&1 
     267    ncatted -O -h -a creation_date,global,o,c,"${yyyymmdd}" ${filein} >> ${log} 2>&1 
     268    status=${?} 
     269    if [ ${status} -ne 0 ] 
     270    then 
     271        echo "eee : pb with ncatted" >> ${log} 2>&1 
     272        ncdump -h ${filein} >> ${log} 2>&1 
     273        exit 1 
     274    fi 
     275    unset status 
     276done 
     277unset filein 
    379278# 
    380279# cleanning 
    381 #++rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 
    382 #++rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 
    383 #++rm -f ${PROJECT_OD}/${fileout}_bad_timerange 2> /dev/null 
    384 # 
    385 unset fileout 
    386 unset filein 
     280#++ 
    387281#++ set 
    388282# end 
  • trunk/src/change_time_range.sh

    r199 r201  
    33#+ 
    44# 
    5 # .. program:: add_19890101.sh 
    6 # 
    7 # .. _add_19890101.sh: 
    8 # 
    9 # ================== 
    10 # add_19890101.sh  
    11 # ================== 
     5# .. program:: change_time_range.sh 
     6# 
     7# .. _change_time_range.sh: 
     8# 
     9# ======================= 
     10# change_time_range.sh 
     11# ======================= 
    1212# 
    1313# SYNOPSIS 
     
    1616# :: 
    1717# 
    18 #  $ add_19890101.sh [--debug] 
     18#  $ change_time_range.sh [--debug] -b yyyymmddb -e yyyymmdde -d directory 
    1919# 
    2020# DESCRIPTION 
     
    2525#    If this option is set, :samp:`ncdump -v tt` will be added to log file 
    2626# 
    27 # Artificialy add 19890101 to lwr interpolated on OAFLUX grid dataset. 
    28 # 
    29 # :file:`${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc` 
    30 # containing  
    31 # ++ 
    32 # has been produced by 
    33 # :func:`interp_erai_lwr`. 
    34 # repeat the first time step in  
    35 # 
    36 # File :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 
    37 # will be written 
    38 # if this file not already exists. 
    39 # 
    40 # This output file  
    41 # :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 
    42 # must be processed after by  
    43 # :func:`lwr_correction_ncdf`. 
    44 #  
    45 # Log file is written on :file:`${PROJECT_LOG}/add_19890101.sh.log.{YYYYMMDDTHHMMSSZ}` 
     27# .. option:: -b <yyyymmddb> 
     28# 
     29#    first date to be written in the global attributes time_range 
     30# 
     31# .. option:: -e <yyyymmdde> 
     32# 
     33#    second date to be written in the global attributes time_range 
     34# 
     35# .. option:: -d <dirin> 
     36# 
     37#    base location of the dataset to be changed 
     38# 
     39# Modify time_range global attribute if the dataset based under dirin parameter 
     40# 
     41# Log file is written on :file:`${PROJECT_LOG}/change_time_range.sh.log.{YYYYMMDDTHHMMSSZ}` 
    4642# 
    4743#     .. graphviz:: 
    4844# 
    49 #        digraph add_19890101 { 
    50 # 
    51 #           filein [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc"]; 
    52 #           fileout [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc"]; 
    53 # 
    54 #           add_19890101 [shape=box, 
     45#        digraph change_time_range { 
     46# 
     47#           change_time_range [shape=box, 
    5548#           fontname=Courier, 
    5649#           color=blue, 
    57 #           URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/add_19890101.sh", 
    58 #           label="${PROJECT}/src/add_19890101.sh"]; 
    59 # 
    60 #           {filein} -> {add_19890101} -> {fileout} 
     50#           URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/change_time_range.sh", 
     51#           label="${PROJECT}/src/change_time_range.sh"]; 
     52# 
     53#           {filein} -> {change_time_range} -> {filein} 
    6154# 
    6255#       } 
     
    6558# ======== 
    6659# 
    67 # To produce $PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc  
    68 # from $PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc :: 
    69 # 
    70 #  $ add_19890101.sh --debug 
     60# To modify files under /usr/lodyc/incas/fplod/tropflux_d/to_be_published/:: 
     61# 
     62#  $ change_time_range.sh -d /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ -b 19790101 -e 20130331 
    7163# 
    7264# And look at log file with :: 
    7365# 
    74 #  $ tlogd.sh add_19890101 
    75 # 
    76 # and of course on files in ${PROJECT_OD}. 
     66#  $ tlogd.sh change_time_range 
    7767# 
    7868# TIPS 
    7969# ==== 
    8070# 
    81 # To see the 3 first time values in input and files:: 
    82 # 
    83 #  $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc  
    84 #    14246.0000000005 
    85 #    14247.0000000005 
    86 #    14248.0000000005 
    87 # 
    88 #  $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc  
    89 #    14245.0000000005 
    90 #    14246.0000000005 
    91 #    14247.0000000005 
    92 # 
    93 # while : 
    94 # 
    95 #   $ idl 
    96 #   print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) 
    97 #   14245.500 
    98 #   print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) +1 
    99 #   14246.500 
    100 # 
    10171# SEE ALSO 
    10272# ======== 
    10373# 
     74# :ref:`updatedata` 
     75# 
    10476# :ref:`project_profile.sh` 
    10577# 
    106 # :func:`ncks <nco:ncks>` 
    107 # :func:`ncrcat <nco:ncrcat>` 
    10878# :func:`ncatted <nco:ncatted>` 
    109 # :func:`ncap2 <nco:ncap2>` 
    110 # 
    111 # :ref:`lwr_correction_ncdf.pro` 
    11279# 
    11380# TODO 
     
    11683# coding rules 
    11784# 
    118 # won't it be more conveniant to add this articifial time step in  
    119 # the IDL creation tool interp_erai_lwr_1989_2009.pro. 
    120 # It will allow to avoid a shell interruption in the IDL processing chain. 
    121 # 
    122 # improve global attributes with ... creation_date and so on 
    123 # 
    124 # check if the first time step is really 19890102 in the input file 
    125 # 
    126 # change tt to ttt to complete time correction 
    127 # 
    128 # remove hard coded end time 
    129 # 
    130 # get rid of this long mesage on cratos:: 
    131 # 
    132 #    ncrcat: WARNING Variable lwr has attribute "missing_value" but not "_FillValue". To  
    133 #    comply with netCDF conventions, NCO ignores values that equal the _FillValue attribu 
    134 #    te when performing arithmetic. Confusingly, values equal to the missing_value should 
    135 #     also be neglected. However, it is tedious and (possibly) computationally expensive  
    136 #    to check each value against multiple missing values during arithmetic on large varia 
    137 #    bles. So NCO thinks that processing variables with a "missing_value" attribute and n 
    138 #    o "_FillValue" attribute may produce undesired arithmetic results (i.e., where value 
    139 #    s that were intended to be neglected were not, in fact, neglected). We suggest you r 
    140 #    ename all "missing_value" attributes to "_FillValue" or include both "missing_value" 
    141 #     and "_FillValue" attributes (with the _same values_) for all variables that have ei 
    142 #    ther attribute. Because it is long, this message is only printed once per operator e 
    143 #    ven though multiple variables may have the same attribute configuration. More inform 
    144 #    ation on missing values is given at: 
    145 #    http://nco.sf.net/nco.html#mss_val 
    146 #    Examples of renaming attributes are at: 
    147 #    http://nco.sf.net/nco.html#xmp_ncrename 
    148 # 
    149 # ... on a déjà une idee de la solution : convention CF et meilleure gestion des 
    150 # valeurs manquantes dans les softs IDL !!! 
    151 # 
     85# overwrite : may be an other option to avoid scratch 
     86# 
     87# debug not used 
     88# 
     89# terminology march2013 non normative 
     90# 
     91# build time period according to time in the file instead of parameter 
    15292# 
    15393# EVOLUTIONS 
     
    15898# $URL$ 
    15999# 
    160 # - fplod 20110819T091143Z aedon.locean-ipsl.upmc.fr (Darwin) 
    161 # 
    162 #   * creation from pk email 20110811 "lwr first timestep" 
    163 #   * include time correction and timerange modificaton 
     100# - fplod 20130726T112836Z cratos.locean-ipsl.upmc.fr (Linux) 
     101# 
     102#   * creation 
    164103# 
    165104#- 
     
    191130log_date=$(date -u +"%Y%m%dT%H%M%SZ") 
    192131# 
    193 usage=" Usage : ${command} [--debug]" 
     132usage=" Usage : ${command} [--debug] -d dirin -b yyyymmddb -e yyyymmdde" 
    194133# 
    195134hostname=$(hostname) 
     
    209148do 
    210149   case ${1} in 
     150      -b) 
     151         yyyymmddb=${2} 
     152         shift 
     153      ;; 
     154      -e) 
     155         yyyymmdde=${2} 
     156         shift 
     157      ;; 
     158      -d) 
     159         dirin=${2} 
     160         shift 
     161      ;; 
    211162      --debug) 
    212163         debug=1 
     
    224175# 
    225176# check parameters 
    226 # N.A. 
    227 # 
    228 tool=ncrcat 
    229 type ${tool} 1> /dev/null 2>&1 
    230 status=${?} 
    231 if [ ${status} -ne 0 ] 
    232 then 
    233    echo "${command} : eee : tool ${tool} not found" 
    234    exit 1 
    235 fi 
    236 unset status 
    237 unset tool 
    238 # 
    239 tool=ncks 
    240 type ${tool} 1> /dev/null 2>&1 
    241 status=${?} 
    242 if [ ${status} -ne 0 ] 
    243 then 
    244    echo "${command} : eee : tool ${tool} not found" 
    245    exit 1 
    246 fi 
    247 unset status 
    248 unset tool 
     177if [ ! -d ${dirin} ] 
     178then 
     179   echo " eee : ${dirin} not found" 
     180   exit 1 
     181fi 
     182# 
     183# check for permission on dirin 
     184if [ ! -x ${dirin} ] 
     185then 
     186   echo " eee : ${dirin} not reachable" 
     187   exit 1 
     188fi 
     189#++ check yyyymmddb and yyyymmddb validity 
    249190# 
    250191tool=ncatted 
     
    253194if [ ${status} -ne 0 ] 
    254195then 
    255    echo "${command} : eee : tool ${tool} not found" 
     196   echo "${command} : ${LINENO} : eee : tool ${tool} not found" 
    256197   exit 1 
    257198fi 
     
    262203if [ "${PROJECT_LOG}" = "" ] 
    263204then 
    264    echo "${command} : eee : \${PROJECT_LOG} not defined" 
     205   echo "${command} : ${LINENO} : eee : \${PROJECT_LOG} not defined" 
    265206   exit 1 
    266207fi 
     
    297238echo "" 1>> ${log} 
    298239echo "[Parameters]" 1>> ${log} 
    299 echo "N.A." 1>> ${log} 
     240echo "dirin=${dirin}" 1>> ${log} 
     241echo "yyyymmddb=${yyyymmddb}" 1>> ${log} 
     242echo "yyyymmdde=${yyyymmdde}" 1>> ${log} 
    300243echo "" 1>> ${log} 
    301244# 
    302 filein=${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc 
    303 if [ ! -f ${filein} ] 
    304 then 
    305     echo " eee : ${filein} not found" 1>> ${log} 
    306     exit 1  
    307 fi 
    308 fileout=${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc 
    309 if [ -f ${fileout} ] 
    310 then 
    311       echo "eee : ${fileout} already exists" 1>>${log} 
    312       exit 1 
    313 fi 
    314 # 
    315 rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 
    316 rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 
    317 # 
    318 # extraction of the first time step 19890102 
    319 ncks -h -d time,0,0 ${filein} ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    320 status=${?} 
    321 if [ ${status} -ne 0 ] 
    322 then 
    323     echo "eee : pb with ncks" >> ${log} 2>&1 
    324     ncdump -h ${filein} >> ${log} 2>&1 
    325     exit 1 
    326 fi 
    327 unset status 
    328 # 
    329 # ++ check if tt[0] is really 19890102 
    330 # 
    331 # substract 1 to tt to simulate 19890101 
    332 ncap2 -h -s "tt=tt-1" \ 
    333     ${PROJECT_OD}/erai_lwr_step19890102.nc \ 
    334     ${PROJECT_OD}/erai_lwr_step19890101.nc 
    335 status=${?} 
    336 if [ ${status} -ne 0 ]  
    337 then 
    338     echo "eee : pb with ncap2" >> ${log} 2>&1 
    339     ncdump -h ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    340     exit 1 
    341 fi   
    342 unset status 
    343 ls -l ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 
    344 ls -l ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 
    345 # 
    346 # ++ check if tt[0] is really 19890101 
    347 #  
    348 rm ${fileout}_bad_timerange 2> /dev/null 
    349 ncrcat -h \ 
    350        ${PROJECT_OD}/erai_lwr_step19890101.nc \ 
    351        ${filein} \ 
    352        ${fileout}_bad_timerange >> ${log} 2>&1 
    353 status=${?} 
    354 if [ ${status} -ne 0 ]  
    355 then 
    356     echo "eee : pb with ncrcat" >> ${log} 2>&1 
    357     ncdump -h ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 
    358     ncdump -h ${filein} >> ${log} 2>&1 
    359     exit 1 
    360 fi   
    361 unset status 
    362 # 
    363 ncatted -h -a timerange,global,o,c,"19890101 - 20091231" \ 
    364        ${fileout}_bad_timerange \ 
    365        ${fileout} >> ${log} 2>&1 
    366 status=${?} 
    367 if [ ${status} -ne 0 ] 
    368 then 
    369     echo "eee : pb with ncatted" >> ${log} 2>&1 
    370     ncdump -h ${fileout}_bad_timerange >> ${log} 2>&1 
    371     exit 1 
    372 fi  
    373 unset status 
    374 # 
    375 if [ ${debug} -eq 1 ] 
    376 then 
    377     ncdump -v tt ${fileout} >> ${log} 2>&1 
    378 fi 
     245# build list files 
     246varlist="lhf lwr netflux q2m shf sst swr t2m tau taux tauy ws" 
     247period_list="daily monthly" 
     248list_filein="" 
     249for var in $varlist 
     250do 
     251     for period in ${period_list} 
     252     do 
     253         if [ "${period}" == "daily" ] 
     254         then 
     255            suffix=1d 
     256         else 
     257            suffix=1m 
     258         fi 
     259         filein=${dirin}/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 
     260         if [ ! -f ${filein} ] 
     261         then 
     262             echo " eee : ${filein} not found" 1>> ${log} 
     263             exit 1 
     264         else 
     265             list_filein=${list_filein}" ${filein}" 
     266         fi 
     267     done 
     268     unset period 
     269done 
     270unset period_list 
     271unset var 
     272unset varlist 
     273# 
     274# build time period 
     275datesec=$(${date_cmd} -u -d "${yyyymmddb}" +%s | awk '{printf "%10.10d",$1}') 
     276yyyyb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%Y) 
     277mmb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%m) 
     278ddb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%d) 
     279unset datesec 
     280datesec=$(${date_cmd} -u -d "${yyyymmdde}" +%s | awk '{printf "%10.10d",$1}') 
     281yyyye=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%Y) 
     282mme=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%m) 
     283dde=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%d) 
     284unset datesec 
     285time_ranged="${yyyymmddb} - ${yyyymmdde}" 
     286time_rangem="${yyyyb}${mmb}15 - ${yyyye}${mme}15" 
     287# 
     288# new time_range attribute 
     289for filein in ${list_filein} 
     290do 
     291    echo "iii : change time_range in ${filein}" >> ${log} 2>&1 
     292    period=$(basename $(dirname $filein)) 
     293    if [ "${period}" == "daily" ] 
     294    then 
     295       time_range=${time_ranged} 
     296    else 
     297       time_range=${time_rangem} 
     298    fi 
     299    ncatted -O -h -a time_range,global,o,c,"${time_range}" ${filein} >> ${log} 2>&1 
     300    status=${?} 
     301    if [ ${status} -ne 0 ] 
     302    then 
     303        echo "eee : pb with ncatted" >> ${log} 2>&1 
     304        ncdump -h ${filein} >> ${log} 2>&1 
     305        exit 1 
     306    fi 
     307    unset status 
     308    ncdump -h ${filein}  | grep time_range >> ${log} 2>&1 
     309done 
     310unset filein 
    379311# 
    380312# cleanning 
    381 #++rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 
    382 #++rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 
    383 #++rm -f ${PROJECT_OD}/${fileout}_bad_timerange 2> /dev/null 
    384 # 
    385 unset fileout 
    386 unset filein 
     313#++ 
    387314#++ set 
    388315# end 
Note: See TracChangeset for help on using the changeset viewer.