- Timestamp:
- 07/26/13 16:21:43 (11 years ago)
- Location:
- trunk
- Files:
-
- 1 added
- 1 edited
- 3 copied
Legend:
- Unmodified
- Added
- Removed
-
trunk/docs/docs_dev/source/guides/updatedata.rst
r196 r201 9 9 .. ======== 10 10 .. 11 .. ref:`updatedata`12 11 .. ref:`builddoc` 13 12 .. … … 37 36 .. 38 37 .. $URL$ 38 .. 39 .. - fplod 20130725T141747Z cratos.locean-ipsl.upmc.fr (Linux) 40 .. 41 .. * detailed specific task for 20130715 update 39 42 .. 40 43 .. - fplod 20121227T192052Z cratos.locean-ipsl.upmc.fr (Linux) … … 87 90 - mettre ce nouveau jeu de données à disposition sous http://www.locean-ipsl.upmc.fr/~tropflux/data/ 88 91 - prévenir les amis de TropFlux (cf. :ref:`advertise`) 92 93 .. note:: 94 95 see detailed and specific tasks for each update below. 89 96 90 97 .. _newversion: … … 229 236 230 237 $ diskext="/Volumes/LCerbere" 238 $ mkdir ${diskext} 231 239 $ sshfs fplod@cerbere:/usr/lodyc/incas/fplod ${diskext} -odefer_permissions -o sshfs_debug 232 240 … … 240 248 Une mise à jour doit être envoyée aux contacts dans les instutions qui mentionne notre jeu de données (NIO). 241 249 250 251 .. toctree:: 252 :maxdepth: 0 253 :glob: 254 255 updatedata/updatedata_20130725 -
trunk/docs/docs_dev/source/guides/updatedata/updatedata_20130725.rst
r199 r201 4 4 .. =========== 5 5 .. 6 .. procedure to update data on website 6 .. procedure to update data on website 20130725 7 7 .. 8 8 .. SEE ALSO … … 10 10 .. 11 11 .. ref:`updatedata` 12 .. ref:`builddoc`13 12 .. 14 13 .. TODO 15 14 .. ===== 16 15 .. 17 .. en fait la comparaison devra être fait aussi AVANT le démarage de cette procédure18 ..19 .. translation20 ..21 .. comprendre les embrouilles ftp, sftp ncftp www.locean-ipsl.upmc.fr22 ..23 .. mettre les sites amis comme le NIO à jour24 ..25 .. définir les outils de comparaison, écrire les outils en question26 ..27 .. tenir compte des images du site dans cette procédure28 ..29 .. tester la mise à jour de Readme.rst30 ..31 .. utilisation du .htaccess32 ..33 16 .. EVOLUTIONS 34 17 .. ========== … … 38 21 .. $URL$ 39 22 .. 40 .. - fplod 20121227T192052Z cratos.locean-ipsl.upmc.fr (Linux) 41 .. 42 .. * no more need to concatenate /net/adonis/usr/adonis/varclim/tropflux_v2 43 .. : jv made it on 20121221 44 .. 45 .. - fplod 20121121T151409Z cratos.locean-ipsl.upmc.fr (Linux) 46 .. 47 .. * update from /net/adonis/usr/adonis/varclim/tropflux_v2 48 .. 2 datasets : 1979-2011 and 201201-201206 49 .. to be concatenated 50 .. 51 .. - fplod 20120307 52 .. 53 .. * README.rst was renamed to Readme.rst because of Apache policy 54 .. 55 .. - fplod 20120229 56 .. 57 .. * associate README.rst to datafiles 58 .. 59 .. - fplod 20120210 60 .. 61 .. * missing an ftp command 62 .. 63 .. - fplod 20120209 64 .. 65 .. * creation (draft) with v0 to v1 action 23 .. - fplod 20130725T141747Z cratos.locean-ipsl.upmc.fr (Linux) 24 .. 25 .. * creation 66 26 .. 67 27 .. - … … 70 30 .. |project_id| replace:: tropflux 71 31 72 .. _updatedata: 73 74 Update data 75 =========== 76 77 On suppose ici que : 78 79 - un nouveau jeu de données est disponible dans un répertoire accessible à partir de cratos.locean-ipsl.upmc.fr (pour faire tourner les outils de validation/comparaison) et à partir du poste de travail (mac) de Françoise Pinsard (pour assurer la mise à jour du serveur de données). 80 - (bientôt) des nouvelles images associées sont disponibles dans les mêmes conditions 81 82 Le but du jeu de la procédure décrite ici est de : 83 84 - conserver le jeu de données actuel (cf. :ref:`saveold`) 85 - conserver le jeu de données nouveau (cf. :ref:`copynew1` and :ref:`copynew2`) 86 - comparer les deux jeux de données (cf. :ref:`compareversion`) 87 - mettre ce nouveau jeu de données à disposition sous http://www.locean-ipsl.upmc.fr/~tropflux/data/ 88 - prévenir les amis de TropFlux (cf. :ref:`advertise`) 89 90 .. _newversion: 91 92 It must be known if the new dataset is a new version of TropFlux product. 93 94 If so, the new version must be described in :file:`changes.rst`. 95 (see also :ref:`builddoc` to synchronize dataset and documentation). 96 97 If a new field is provided it must be described in :file:`data_out_website.rst` 98 99 :file:`Readme.rst` may also be updated. 100 101 .. warning:: 102 103 embrouille entre les machines (cratos vs mon poste de travail) et les outils (ftp, ncftp, sftp) : on arrive pas toujours à trouver le répertoire |project_id| après la connexion. 104 105 En conséquence, toutes les commandes ftp sont faites sur le poste de travail (mac) de Françoise Pinsard. 106 107 .. note:: 108 109 Here is the sequence of commands to be run to create 110 the directory equivalent to the URL http://www.locean-ipsl.upmc.fr/~tropflux/images/ 111 112 .. parsed-literal:: 113 114 fplod@sonmac$ ftp www.locean-ipsl.upmc.fr 115 ftp> cd |project_id| 116 ftp> mkdir images 117 118 La mise à disposition des fichiers de données et des images sur l'URL 119 http://www.locean-ipsl.upmc.fr/tropflux/ 120 est réservée à Françoise Pinsard pour des raisons de protections de fichiers et 121 de politique de login. 32 .. _updatedata_20130725: 33 34 Update data 20130725 35 ==================== 36 37 Here are details tasks for the update data 20130725 derived from :ref:`updatedata`. 38 39 .. _newversion_20130725: 40 41 Only time coverage extension from July 2012 to March 2013 42 43 :file:`${PROJECT}/website/data/changes.rst` and :file:`${PROJECT}/adm/website/Readme.rst` are updated. 122 44 123 45 .. _saveold: … … 128 50 .. parsed-literal:: 129 51 130 $ datemodif="201 20209"52 $ datemodif="20130725" 131 53 132 54 $ export datemodif … … 140 62 $ mv ./monthly obsolete_${datemodif} 141 63 142 .. _copynew1 :64 .. _copynew1_20130725: 143 65 144 66 Copy new version (phase 1) … … 147 69 .. parsed-literal:: 148 70 149 $ newdir='/net/adonis/usr/adonis/varclim/tropflux_v1' 71 $ mkdir /usr/lodyc/incas/fplod/tropflux_d/to_be_published 72 $ newdir='/net/adonis/usr/adonis/varclim/tropflux_v2' 150 73 $ export newdir 151 $ svn export svn+ssh://pinsard@forge.ipsl.jussieu.fr/ipsl/forge/projets/tropflux/svn/trunk/adm/website/Readme.rst \ 152 /usr/lodyc/incas/fplod/tropflux_d/to_be_published/Readme.rst 74 75 Is there enought space ? 76 $ du -sh ${newdir}/daily 77 12G /net/adonis/usr/adonis/varclim/tropflux_v2/daily 78 $ du -sh ${newdir}/monthly 79 396M /net/adonis/usr/adonis/varclim/tropflux_v2/monthly 80 $ df -h /usr/lodyc/incas/fplod/tropflux_d/to_be_published 81 Filesystem Size Used Avail Use% Mounted on 82 fslodyc:/usr/.lodyc 1.6T 1.6T 20G 99% /.autofs/lodyc 83 84 20GB vs 12.4 GB = yes 85 86 $ cp ${PROJECT}/adm/website/Readme.rst /usr/lodyc/incas/fplod/tropflux_d/to_be_published/Readme.rst 153 87 $ cp -r ${newdir}/daily /usr/lodyc/incas/fplod/tropflux_d/to_be_published 154 88 $ cp -r ${newdir}/monthly /usr/lodyc/incas/fplod/tropflux_d/to_be_published 155 89 156 .. _compareversion :90 .. _compareversion_20130725: 157 91 158 92 Compare versions … … 167 101 $ ++compare.sh -o obsolete_${datemodif} -n /usr/lodyc/incas/fplod/tropflux_d/to_be_published 168 102 103 mkdir -p /usr/work/incas/fplod/tropflux_d//compare_update/ 104 varlist="lhf lwr netflux q2m shf swr t2m tau taux tauy ws" 105 period_list="daily monthly" 106 lastyear=2012 107 for var in $varlist 108 do 109 for period in ${period_list} 110 do 111 if [ "${period}" == "daily" ] 112 then 113 suffix=1d 114 else 115 suffix=1m 116 fi 117 newfile=/usr/lodyc/incas/fplod/tropflux_d/to_be_published/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 118 newfile_oldtimecoverage=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastdate}.nc 119 # extract old time coverage in the new file 120 nt=$(ncdump -h $oldfile | grep "time = UNLIMITED ;" | sed -e "s@\(time = UNLIMITED ; // (\)\(.*\)\( currently)\)@\2@") 121 slabtime=$(printf "time,0,%d" $((${nt} -1 ))) 122 ncks -d ${slabtime} -O -o ${newfile_oldtimecoverage} ${newfile} 123 oldfile=/usr/lodyc/incas/fplod/tropflux_d/obsolete_${datemodif}/${period}/${var}_tropflux_${suffix}_1979_${lastyear}.nc 124 oldfiledump=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastyear}.nc_ncdump 125 cdo diff ${newfile_oldtimecoverage} ${oldfile} 126 read a 127 # we must see "0 of $nt records differ" 128 ncdump -h ${newfile_oldtimecoverage} > ${newfile_oldtimecoverage}_ncdump 129 ncdump -h ${oldfile} > ${oldfiledump} 130 diff ${newfile_oldtimecoverage}_ncdump ${oldfiledump} 131 # we must see a difference in creation date and coverage 132 read a 133 rm ${newfile_oldtimecoverage}_ncdump 134 rm ${oldfiledump} 135 unset suffix 136 done 137 unset period 138 unset period_list 139 done 140 unset var 141 unset varlist 142 rm -rf /usr/work/incas/fplod/tropflux_d//compare_update/ 143 144 log=${PROJECT_LOG}/compare.log 145 script ${PROJECT_LOG}/compare.log 146 compare.sh 147 exit 148 149 Check for results. 150 151 First check expected equality for data: 152 153 :: 154 155 grep "records differ" ${log} 156 157 If only "0 of" begin all lines, its ok. Otherwise look inside the log file to 158 locate troubles. 159 160 Second, check for differences on global attributes 161 162 :: 163 164 grep "^>" ${log} | grep creation_date 165 166 If nothing appears, you must modified this attributes. see :ref:`change_creation_date.sh` ++ 167 168 169 grep "^>" ${log} | grep time_range 170 171 If nothing appears, you must modified this attributes. see :ref:`change_timerange.sh` ++ 172 173 Clean log file:: 174 175 rm ${log} 176 177 If modifications has been made, re-run comparaison before next step 178 169 179 Once comparison is done and PI happy, 170 180 ++ cleanning or modify newdir 171 181 172 .. _copynew2 :182 .. _copynew2_20130725: 173 183 174 184 Copy new version (phase 2) … … 185 195 186 196 187 .. _desinstalldata :197 .. _desinstalldata_20130725: 188 198 189 199 Remove old version from the website … … 199 209 ftp> mdelete monthly/* 200 210 201 .. _installdata :211 .. _installdata_20130725: 202 212 203 213 Installation of the new version … … 229 239 230 240 $ diskext="/Volumes/LCerbere" 241 $ mkdir ${diskext} 231 242 $ sshfs fplod@cerbere:/usr/lodyc/incas/fplod ${diskext} -odefer_permissions -o sshfs_debug 232 243 233 .. _advertise: 234 235 Advertise TropFlux friends 236 -------------------------- 237 238 A email must be send to TropFlux list to warn about the new dataset. 239 240 Une mise à jour doit être envoyée aux contacts dans les instutions qui mentionne notre jeu de données (NIO). 241 244 245 et template pour NEW 246 247 et build doc et install doc 248 249 (see also :ref:`builddoc` to synchronize dataset and documentation). 250 251 252 et svn commit ppur ce fichier et tous ceux mentionnes ici 253 rm /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ 254 -
trunk/src/change_creation_date.sh
r199 r201 3 3 #+ 4 4 # 5 # .. program:: add_19890101.sh6 # 7 # .. _ add_19890101.sh:8 # 9 # ================== 10 # add_19890101.sh11 # ================== 5 # .. program:: change_creation_date.sh 6 # 7 # .. _change_creation_date.sh: 8 # 9 # ======================= 10 # change_creation_date.sh 11 # ======================= 12 12 # 13 13 # SYNOPSIS … … 16 16 # :: 17 17 # 18 # $ add_19890101.sh [--debug]18 # $ change_creation_date.sh [--debug] -c yyyymmdd -d directory 19 19 # 20 20 # DESCRIPTION … … 25 25 # If this option is set, :samp:`ncdump -v tt` will be added to log file 26 26 # 27 # Artificialy add 19890101 to lwr interpolated on OAFLUX grid dataset. 28 # 29 # :file:`${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc` 30 # containing 31 # ++ 32 # has been produced by 33 # :func:`interp_erai_lwr`. 34 # repeat the first time step in 35 # 36 # File :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 37 # will be written 38 # if this file not already exists. 39 # 40 # This output file 41 # :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 42 # must be processed after by 43 # :func:`lwr_correction_ncdf`. 44 # 45 # Log file is written on :file:`${PROJECT_LOG}/add_19890101.sh.log.{YYYYMMDDTHHMMSSZ}` 27 # .. option:: -c <yyyymmdd> 28 # 29 # date to be written in the global attributes creation_date 30 # 31 # .. option:: -d <dirin> 32 # 33 # base location of the dataset to be changed 34 # 35 # Modify creation_date global attribute if the dataset based under dirin parameter 36 # 37 # Log file is written on :file:`${PROJECT_LOG}/change_creation_date.sh.log.{YYYYMMDDTHHMMSSZ}` 46 38 # 47 39 # .. graphviz:: 48 40 # 49 # digraph add_19890101 { 50 # 51 # filein [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc"]; 52 # fileout [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc"]; 53 # 54 # add_19890101 [shape=box, 41 # digraph change_creation_date { 42 # 43 # change_creation_date [shape=box, 55 44 # fontname=Courier, 56 45 # color=blue, 57 # URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/ add_19890101.sh",58 # label="${PROJECT}/src/ add_19890101.sh"];59 # 60 # {filein} -> { add_19890101} -> {fileout}46 # URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/change_creation_date.sh", 47 # label="${PROJECT}/src/change_creation_date.sh"]; 48 # 49 # {filein} -> {change_creation_date} -> {filein} 61 50 # 62 51 # } … … 65 54 # ======== 66 55 # 67 # To produce $PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc 68 # from $PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc :: 69 # 70 # $ add_19890101.sh --debug 56 # To modify files under /usr/lodyc/incas/fplod/tropflux_d/to_be_published/:: 57 # 58 # $ change_creation_date.sh -d /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ -c 20130725 71 59 # 72 60 # And look at log file with :: 73 61 # 74 # $ tlogd.sh add_19890101 75 # 76 # and of course on files in ${PROJECT_OD}. 62 # $ tlogd.sh change_creation_date 77 63 # 78 64 # TIPS 79 65 # ==== 80 66 # 81 # To see the 3 first time values in input and files::82 #83 # $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc84 # 14246.000000000585 # 14247.000000000586 # 14248.000000000587 #88 # $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc89 # 14245.000000000590 # 14246.000000000591 # 14247.000000000592 #93 # while :94 #95 # $ idl96 # print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0)97 # 14245.50098 # print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) +199 # 14246.500100 #101 67 # SEE ALSO 102 68 # ======== 103 69 # 70 # :ref:`updatedata` 71 # 104 72 # :ref:`project_profile.sh` 105 73 # 106 # :func:`ncks <nco:ncks>`107 # :func:`ncrcat <nco:ncrcat>`108 74 # :func:`ncatted <nco:ncatted>` 109 # :func:`ncap2 <nco:ncap2>`110 #111 # :ref:`lwr_correction_ncdf.pro`112 75 # 113 76 # TODO … … 116 79 # coding rules 117 80 # 118 # won't it be more conveniant to add this articifial time step in 119 # the IDL creation tool interp_erai_lwr_1989_2009.pro. 120 # It will allow to avoid a shell interruption in the IDL processing chain. 121 # 122 # improve global attributes with ... creation_date and so on 123 # 124 # check if the first time step is really 19890102 in the input file 125 # 126 # change tt to ttt to complete time correction 127 # 128 # remove hard coded end time 129 # 130 # get rid of this long mesage on cratos:: 131 # 132 # ncrcat: WARNING Variable lwr has attribute "missing_value" but not "_FillValue". To 133 # comply with netCDF conventions, NCO ignores values that equal the _FillValue attribu 134 # te when performing arithmetic. Confusingly, values equal to the missing_value should 135 # also be neglected. However, it is tedious and (possibly) computationally expensive 136 # to check each value against multiple missing values during arithmetic on large varia 137 # bles. So NCO thinks that processing variables with a "missing_value" attribute and n 138 # o "_FillValue" attribute may produce undesired arithmetic results (i.e., where value 139 # s that were intended to be neglected were not, in fact, neglected). We suggest you r 140 # ename all "missing_value" attributes to "_FillValue" or include both "missing_value" 141 # and "_FillValue" attributes (with the _same values_) for all variables that have ei 142 # ther attribute. Because it is long, this message is only printed once per operator e 143 # ven though multiple variables may have the same attribute configuration. More inform 144 # ation on missing values is given at: 145 # http://nco.sf.net/nco.html#mss_val 146 # Examples of renaming attributes are at: 147 # http://nco.sf.net/nco.html#xmp_ncrename 148 # 149 # ... on a déjà une idee de la solution : convention CF et meilleure gestion des 150 # valeurs manquantes dans les softs IDL !!! 151 # 81 # overwrite : may be an other option to avoid scratch 82 # 83 # debug not used 84 # 85 # terminology march2013 non normative 152 86 # 153 87 # EVOLUTIONS … … 158 92 # $URL$ 159 93 # 160 # - fplod 20110819T091143Z aedon.locean-ipsl.upmc.fr (Darwin) 161 # 162 # * creation from pk email 20110811 "lwr first timestep" 163 # * include time correction and timerange modificaton 94 # - fplod 20130726T112836Z cratos.locean-ipsl.upmc.fr (Linux) 95 # 96 # * creation 164 97 # 165 98 #- … … 191 124 log_date=$(date -u +"%Y%m%dT%H%M%SZ") 192 125 # 193 usage=" Usage : ${command} [--debug] "126 usage=" Usage : ${command} [--debug] -d dirin -c yyyymmdd" 194 127 # 195 128 hostname=$(hostname) … … 209 142 do 210 143 case ${1} in 144 -c) 145 yyyymmdd=${2} 146 shift 147 ;; 148 -d) 149 dirin=${2} 150 shift 151 ;; 211 152 --debug) 212 153 debug=1 … … 224 165 # 225 166 # check parameters 226 # N.A. 227 # 228 tool=ncrcat 229 type ${tool} 1> /dev/null 2>&1 230 status=${?} 231 if [ ${status} -ne 0 ] 232 then 233 echo "${command} : eee : tool ${tool} not found" 234 exit 1 235 fi 236 unset status 237 unset tool 238 # 239 tool=ncks 240 type ${tool} 1> /dev/null 2>&1 241 status=${?} 242 if [ ${status} -ne 0 ] 243 then 244 echo "${command} : eee : tool ${tool} not found" 245 exit 1 246 fi 247 unset status 248 unset tool 167 if [ ! -d ${dirin} ] 168 then 169 echo " eee : ${dirin} not found" 170 exit 1 171 fi 172 # 173 # check for permission on dirin 174 if [ ! -x ${dirin} ] 175 then 176 echo " eee : ${dirin} not reachable" 177 exit 1 178 fi 179 #++ check yyyymmdd validity 249 180 # 250 181 tool=ncatted … … 253 184 if [ ${status} -ne 0 ] 254 185 then 255 echo "${command} : eee : tool ${tool} not found"186 echo "${command} : ${LINENO} : eee : tool ${tool} not found" 256 187 exit 1 257 188 fi … … 262 193 if [ "${PROJECT_LOG}" = "" ] 263 194 then 264 echo "${command} : eee : \${PROJECT_LOG} not defined"195 echo "${command} : ${LINENO} : eee : \${PROJECT_LOG} not defined" 265 196 exit 1 266 197 fi … … 297 228 echo "" 1>> ${log} 298 229 echo "[Parameters]" 1>> ${log} 299 echo "N.A." 1>> ${log} 230 echo "dirin=${dirin}" 1>> ${log} 231 echo "yyyymmdd=${yyyymmdd}" 1>> ${log} 300 232 echo "" 1>> ${log} 301 233 # 302 filein=${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc 303 if [ ! -f ${filein} ] 304 then 305 echo " eee : ${filein} not found" 1>> ${log} 306 exit 1 307 fi 308 fileout=${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc 309 if [ -f ${fileout} ] 310 then 311 echo "eee : ${fileout} already exists" 1>>${log} 312 exit 1 313 fi 314 # 315 rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 316 rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 317 # 318 # extraction of the first time step 19890102 319 ncks -h -d time,0,0 ${filein} ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 320 status=${?} 321 if [ ${status} -ne 0 ] 322 then 323 echo "eee : pb with ncks" >> ${log} 2>&1 324 ncdump -h ${filein} >> ${log} 2>&1 325 exit 1 326 fi 327 unset status 328 # 329 # ++ check if tt[0] is really 19890102 330 # 331 # substract 1 to tt to simulate 19890101 332 ncap2 -h -s "tt=tt-1" \ 333 ${PROJECT_OD}/erai_lwr_step19890102.nc \ 334 ${PROJECT_OD}/erai_lwr_step19890101.nc 335 status=${?} 336 if [ ${status} -ne 0 ] 337 then 338 echo "eee : pb with ncap2" >> ${log} 2>&1 339 ncdump -h ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 340 exit 1 341 fi 342 unset status 343 ls -l ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 344 ls -l ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 345 # 346 # ++ check if tt[0] is really 19890101 347 # 348 rm ${fileout}_bad_timerange 2> /dev/null 349 ncrcat -h \ 350 ${PROJECT_OD}/erai_lwr_step19890101.nc \ 351 ${filein} \ 352 ${fileout}_bad_timerange >> ${log} 2>&1 353 status=${?} 354 if [ ${status} -ne 0 ] 355 then 356 echo "eee : pb with ncrcat" >> ${log} 2>&1 357 ncdump -h ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 358 ncdump -h ${filein} >> ${log} 2>&1 359 exit 1 360 fi 361 unset status 362 # 363 ncatted -h -a timerange,global,o,c,"19890101 - 20091231" \ 364 ${fileout}_bad_timerange \ 365 ${fileout} >> ${log} 2>&1 366 status=${?} 367 if [ ${status} -ne 0 ] 368 then 369 echo "eee : pb with ncatted" >> ${log} 2>&1 370 ncdump -h ${fileout}_bad_timerange >> ${log} 2>&1 371 exit 1 372 fi 373 unset status 374 # 375 if [ ${debug} -eq 1 ] 376 then 377 ncdump -v tt ${fileout} >> ${log} 2>&1 378 fi 234 # build list files 235 varlist="lhf lwr netflux q2m shf sst swr t2m tau taux tauy ws" 236 period_list="daily monthly" 237 list_filein="" 238 for var in $varlist 239 do 240 for period in ${period_list} 241 do 242 if [ "${period}" == "daily" ] 243 then 244 suffix=1d 245 else 246 suffix=1m 247 fi 248 filein=${dirin}/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 249 if [ ! -f ${filein} ] 250 then 251 echo " eee : ${filein} not found" 1>> ${log} 252 exit 1 253 else 254 list_filein=${list_filein}" ${filein}" 255 fi 256 done 257 unset period 258 done 259 unset period_list 260 unset var 261 unset varlist 262 # 263 # new creation_date attribute 264 for filein in ${list_filein} 265 do 266 echo "iii : change creation_date in ${filein}" >> ${log} 2>&1 267 ncatted -O -h -a creation_date,global,o,c,"${yyyymmdd}" ${filein} >> ${log} 2>&1 268 status=${?} 269 if [ ${status} -ne 0 ] 270 then 271 echo "eee : pb with ncatted" >> ${log} 2>&1 272 ncdump -h ${filein} >> ${log} 2>&1 273 exit 1 274 fi 275 unset status 276 done 277 unset filein 379 278 # 380 279 # cleanning 381 #++rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 382 #++rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 383 #++rm -f ${PROJECT_OD}/${fileout}_bad_timerange 2> /dev/null 384 # 385 unset fileout 386 unset filein 280 #++ 387 281 #++ set 388 282 # end -
trunk/src/change_time_range.sh
r199 r201 3 3 #+ 4 4 # 5 # .. program:: add_19890101.sh6 # 7 # .. _ add_19890101.sh:8 # 9 # ================== 10 # add_19890101.sh11 # ================== 5 # .. program:: change_time_range.sh 6 # 7 # .. _change_time_range.sh: 8 # 9 # ======================= 10 # change_time_range.sh 11 # ======================= 12 12 # 13 13 # SYNOPSIS … … 16 16 # :: 17 17 # 18 # $ add_19890101.sh [--debug]18 # $ change_time_range.sh [--debug] -b yyyymmddb -e yyyymmdde -d directory 19 19 # 20 20 # DESCRIPTION … … 25 25 # If this option is set, :samp:`ncdump -v tt` will be added to log file 26 26 # 27 # Artificialy add 19890101 to lwr interpolated on OAFLUX grid dataset. 28 # 29 # :file:`${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc` 30 # containing 31 # ++ 32 # has been produced by 33 # :func:`interp_erai_lwr`. 34 # repeat the first time step in 35 # 36 # File :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 37 # will be written 38 # if this file not already exists. 39 # 40 # This output file 41 # :file:`${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc` 42 # must be processed after by 43 # :func:`lwr_correction_ncdf`. 44 # 45 # Log file is written on :file:`${PROJECT_LOG}/add_19890101.sh.log.{YYYYMMDDTHHMMSSZ}` 27 # .. option:: -b <yyyymmddb> 28 # 29 # first date to be written in the global attributes time_range 30 # 31 # .. option:: -e <yyyymmdde> 32 # 33 # second date to be written in the global attributes time_range 34 # 35 # .. option:: -d <dirin> 36 # 37 # base location of the dataset to be changed 38 # 39 # Modify time_range global attribute if the dataset based under dirin parameter 40 # 41 # Log file is written on :file:`${PROJECT_LOG}/change_time_range.sh.log.{YYYYMMDDTHHMMSSZ}` 46 42 # 47 43 # .. graphviz:: 48 44 # 49 # digraph add_19890101 { 50 # 51 # filein [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc"]; 52 # fileout [shape=ellipse,fontname=Courier,label="$PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc"]; 53 # 54 # add_19890101 [shape=box, 45 # digraph change_time_range { 46 # 47 # change_time_range [shape=box, 55 48 # fontname=Courier, 56 49 # color=blue, 57 # URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/ add_19890101.sh",58 # label="${PROJECT}/src/ add_19890101.sh"];59 # 60 # {filein} -> { add_19890101} -> {fileout}50 # URL="http://forge.ipsl.jussieu.fr/tropflux/browser/trunk/src/change_time_range.sh", 51 # label="${PROJECT}/src/change_time_range.sh"]; 52 # 53 # {filein} -> {change_time_range} -> {filein} 61 54 # 62 55 # } … … 65 58 # ======== 66 59 # 67 # To produce $PROJECT_OD/erai_lwr_19890101_20091231_oafluxgrid.nc 68 # from $PROJECT_OD/erai_lwr_19890102_20091231_oafluxgrid.nc :: 69 # 70 # $ add_19890101.sh --debug 60 # To modify files under /usr/lodyc/incas/fplod/tropflux_d/to_be_published/:: 61 # 62 # $ change_time_range.sh -d /usr/lodyc/incas/fplod/tropflux_d/to_be_published/ -b 19790101 -e 20130331 71 63 # 72 64 # And look at log file with :: 73 65 # 74 # $ tlogd.sh add_19890101 75 # 76 # and of course on files in ${PROJECT_OD}. 66 # $ tlogd.sh change_time_range 77 67 # 78 68 # TIPS 79 69 # ==== 80 70 # 81 # To see the 3 first time values in input and files::82 #83 # $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc84 # 14246.000000000585 # 14247.000000000586 # 14248.000000000587 #88 # $ ncks -s "%16.10f\n" -v tt -d time,0,2 ${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc89 # 14245.000000000590 # 14246.000000000591 # 14247.000000000592 #93 # while :94 #95 # $ idl96 # print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0)97 # 14245.50098 # print, julday(01, 01, 1989,12)-julday(01, 01, 1950,0) +199 # 14246.500100 #101 71 # SEE ALSO 102 72 # ======== 103 73 # 74 # :ref:`updatedata` 75 # 104 76 # :ref:`project_profile.sh` 105 77 # 106 # :func:`ncks <nco:ncks>`107 # :func:`ncrcat <nco:ncrcat>`108 78 # :func:`ncatted <nco:ncatted>` 109 # :func:`ncap2 <nco:ncap2>`110 #111 # :ref:`lwr_correction_ncdf.pro`112 79 # 113 80 # TODO … … 116 83 # coding rules 117 84 # 118 # won't it be more conveniant to add this articifial time step in 119 # the IDL creation tool interp_erai_lwr_1989_2009.pro. 120 # It will allow to avoid a shell interruption in the IDL processing chain. 121 # 122 # improve global attributes with ... creation_date and so on 123 # 124 # check if the first time step is really 19890102 in the input file 125 # 126 # change tt to ttt to complete time correction 127 # 128 # remove hard coded end time 129 # 130 # get rid of this long mesage on cratos:: 131 # 132 # ncrcat: WARNING Variable lwr has attribute "missing_value" but not "_FillValue". To 133 # comply with netCDF conventions, NCO ignores values that equal the _FillValue attribu 134 # te when performing arithmetic. Confusingly, values equal to the missing_value should 135 # also be neglected. However, it is tedious and (possibly) computationally expensive 136 # to check each value against multiple missing values during arithmetic on large varia 137 # bles. So NCO thinks that processing variables with a "missing_value" attribute and n 138 # o "_FillValue" attribute may produce undesired arithmetic results (i.e., where value 139 # s that were intended to be neglected were not, in fact, neglected). We suggest you r 140 # ename all "missing_value" attributes to "_FillValue" or include both "missing_value" 141 # and "_FillValue" attributes (with the _same values_) for all variables that have ei 142 # ther attribute. Because it is long, this message is only printed once per operator e 143 # ven though multiple variables may have the same attribute configuration. More inform 144 # ation on missing values is given at: 145 # http://nco.sf.net/nco.html#mss_val 146 # Examples of renaming attributes are at: 147 # http://nco.sf.net/nco.html#xmp_ncrename 148 # 149 # ... on a déjà une idee de la solution : convention CF et meilleure gestion des 150 # valeurs manquantes dans les softs IDL !!! 151 # 85 # overwrite : may be an other option to avoid scratch 86 # 87 # debug not used 88 # 89 # terminology march2013 non normative 90 # 91 # build time period according to time in the file instead of parameter 152 92 # 153 93 # EVOLUTIONS … … 158 98 # $URL$ 159 99 # 160 # - fplod 20110819T091143Z aedon.locean-ipsl.upmc.fr (Darwin) 161 # 162 # * creation from pk email 20110811 "lwr first timestep" 163 # * include time correction and timerange modificaton 100 # - fplod 20130726T112836Z cratos.locean-ipsl.upmc.fr (Linux) 101 # 102 # * creation 164 103 # 165 104 #- … … 191 130 log_date=$(date -u +"%Y%m%dT%H%M%SZ") 192 131 # 193 usage=" Usage : ${command} [--debug] "132 usage=" Usage : ${command} [--debug] -d dirin -b yyyymmddb -e yyyymmdde" 194 133 # 195 134 hostname=$(hostname) … … 209 148 do 210 149 case ${1} in 150 -b) 151 yyyymmddb=${2} 152 shift 153 ;; 154 -e) 155 yyyymmdde=${2} 156 shift 157 ;; 158 -d) 159 dirin=${2} 160 shift 161 ;; 211 162 --debug) 212 163 debug=1 … … 224 175 # 225 176 # check parameters 226 # N.A. 227 # 228 tool=ncrcat 229 type ${tool} 1> /dev/null 2>&1 230 status=${?} 231 if [ ${status} -ne 0 ] 232 then 233 echo "${command} : eee : tool ${tool} not found" 234 exit 1 235 fi 236 unset status 237 unset tool 238 # 239 tool=ncks 240 type ${tool} 1> /dev/null 2>&1 241 status=${?} 242 if [ ${status} -ne 0 ] 243 then 244 echo "${command} : eee : tool ${tool} not found" 245 exit 1 246 fi 247 unset status 248 unset tool 177 if [ ! -d ${dirin} ] 178 then 179 echo " eee : ${dirin} not found" 180 exit 1 181 fi 182 # 183 # check for permission on dirin 184 if [ ! -x ${dirin} ] 185 then 186 echo " eee : ${dirin} not reachable" 187 exit 1 188 fi 189 #++ check yyyymmddb and yyyymmddb validity 249 190 # 250 191 tool=ncatted … … 253 194 if [ ${status} -ne 0 ] 254 195 then 255 echo "${command} : eee : tool ${tool} not found"196 echo "${command} : ${LINENO} : eee : tool ${tool} not found" 256 197 exit 1 257 198 fi … … 262 203 if [ "${PROJECT_LOG}" = "" ] 263 204 then 264 echo "${command} : eee : \${PROJECT_LOG} not defined"205 echo "${command} : ${LINENO} : eee : \${PROJECT_LOG} not defined" 265 206 exit 1 266 207 fi … … 297 238 echo "" 1>> ${log} 298 239 echo "[Parameters]" 1>> ${log} 299 echo "N.A." 1>> ${log} 240 echo "dirin=${dirin}" 1>> ${log} 241 echo "yyyymmddb=${yyyymmddb}" 1>> ${log} 242 echo "yyyymmdde=${yyyymmdde}" 1>> ${log} 300 243 echo "" 1>> ${log} 301 244 # 302 filein=${PROJECT_OD}/erai_lwr_19890102_20091231_oafluxgrid.nc 303 if [ ! -f ${filein} ] 304 then 305 echo " eee : ${filein} not found" 1>> ${log} 306 exit 1 307 fi 308 fileout=${PROJECT_OD}/erai_lwr_19890101_20091231_oafluxgrid.nc 309 if [ -f ${fileout} ] 310 then 311 echo "eee : ${fileout} already exists" 1>>${log} 312 exit 1 313 fi 314 # 315 rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 316 rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 317 # 318 # extraction of the first time step 19890102 319 ncks -h -d time,0,0 ${filein} ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 320 status=${?} 321 if [ ${status} -ne 0 ] 322 then 323 echo "eee : pb with ncks" >> ${log} 2>&1 324 ncdump -h ${filein} >> ${log} 2>&1 325 exit 1 326 fi 327 unset status 328 # 329 # ++ check if tt[0] is really 19890102 330 # 331 # substract 1 to tt to simulate 19890101 332 ncap2 -h -s "tt=tt-1" \ 333 ${PROJECT_OD}/erai_lwr_step19890102.nc \ 334 ${PROJECT_OD}/erai_lwr_step19890101.nc 335 status=${?} 336 if [ ${status} -ne 0 ] 337 then 338 echo "eee : pb with ncap2" >> ${log} 2>&1 339 ncdump -h ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 340 exit 1 341 fi 342 unset status 343 ls -l ${PROJECT_OD}/erai_lwr_step19890102.nc >> ${log} 2>&1 344 ls -l ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 345 # 346 # ++ check if tt[0] is really 19890101 347 # 348 rm ${fileout}_bad_timerange 2> /dev/null 349 ncrcat -h \ 350 ${PROJECT_OD}/erai_lwr_step19890101.nc \ 351 ${filein} \ 352 ${fileout}_bad_timerange >> ${log} 2>&1 353 status=${?} 354 if [ ${status} -ne 0 ] 355 then 356 echo "eee : pb with ncrcat" >> ${log} 2>&1 357 ncdump -h ${PROJECT_OD}/erai_lwr_step19890101.nc >> ${log} 2>&1 358 ncdump -h ${filein} >> ${log} 2>&1 359 exit 1 360 fi 361 unset status 362 # 363 ncatted -h -a timerange,global,o,c,"19890101 - 20091231" \ 364 ${fileout}_bad_timerange \ 365 ${fileout} >> ${log} 2>&1 366 status=${?} 367 if [ ${status} -ne 0 ] 368 then 369 echo "eee : pb with ncatted" >> ${log} 2>&1 370 ncdump -h ${fileout}_bad_timerange >> ${log} 2>&1 371 exit 1 372 fi 373 unset status 374 # 375 if [ ${debug} -eq 1 ] 376 then 377 ncdump -v tt ${fileout} >> ${log} 2>&1 378 fi 245 # build list files 246 varlist="lhf lwr netflux q2m shf sst swr t2m tau taux tauy ws" 247 period_list="daily monthly" 248 list_filein="" 249 for var in $varlist 250 do 251 for period in ${period_list} 252 do 253 if [ "${period}" == "daily" ] 254 then 255 suffix=1d 256 else 257 suffix=1m 258 fi 259 filein=${dirin}/${period}/${var}_tropflux_${suffix}_1979_march2013.nc 260 if [ ! -f ${filein} ] 261 then 262 echo " eee : ${filein} not found" 1>> ${log} 263 exit 1 264 else 265 list_filein=${list_filein}" ${filein}" 266 fi 267 done 268 unset period 269 done 270 unset period_list 271 unset var 272 unset varlist 273 # 274 # build time period 275 datesec=$(${date_cmd} -u -d "${yyyymmddb}" +%s | awk '{printf "%10.10d",$1}') 276 yyyyb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%Y) 277 mmb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%m) 278 ddb=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%d) 279 unset datesec 280 datesec=$(${date_cmd} -u -d "${yyyymmdde}" +%s | awk '{printf "%10.10d",$1}') 281 yyyye=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%Y) 282 mme=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%m) 283 dde=$(${date_cmd} -u -d "1970-01-01 ${datesec} sec" +%d) 284 unset datesec 285 time_ranged="${yyyymmddb} - ${yyyymmdde}" 286 time_rangem="${yyyyb}${mmb}15 - ${yyyye}${mme}15" 287 # 288 # new time_range attribute 289 for filein in ${list_filein} 290 do 291 echo "iii : change time_range in ${filein}" >> ${log} 2>&1 292 period=$(basename $(dirname $filein)) 293 if [ "${period}" == "daily" ] 294 then 295 time_range=${time_ranged} 296 else 297 time_range=${time_rangem} 298 fi 299 ncatted -O -h -a time_range,global,o,c,"${time_range}" ${filein} >> ${log} 2>&1 300 status=${?} 301 if [ ${status} -ne 0 ] 302 then 303 echo "eee : pb with ncatted" >> ${log} 2>&1 304 ncdump -h ${filein} >> ${log} 2>&1 305 exit 1 306 fi 307 unset status 308 ncdump -h ${filein} | grep time_range >> ${log} 2>&1 309 done 310 unset filein 379 311 # 380 312 # cleanning 381 #++rm -f ${PROJECT_OD}/erai_lwr_step19890101.nc 2> /dev/null 382 #++rm -f ${PROJECT_OD}/erai_lwr_step19890102.nc 2> /dev/null 383 #++rm -f ${PROJECT_OD}/${fileout}_bad_timerange 2> /dev/null 384 # 385 unset fileout 386 unset filein 313 #++ 387 314 #++ set 388 315 # end
Note: See TracChangeset
for help on using the changeset viewer.