.. +
..
.. DESCRIPTION
.. ===========
..
.. procedure to update data on website 20130725
..
.. SEE ALSO
.. ========
..
.. ref:`updatedata`
..
.. TODO
.. =====
..
.. EVOLUTIONS
.. ==========
..
.. $Id$
..
.. $URL$
..
.. - fplod 20130725T141747Z cratos.locean-ipsl.upmc.fr (Linux)
..
..   * creation
..
.. -

.. |project| replace:: TROPFLUX
.. |project_id| replace:: tropflux

.. _updatedata_20130725:

Update data 20130725
====================

Here are details tasks for the update data 20130725 derived from :ref:`updatedata`.

.. _newversion_20130725:

Only time coverage extension from July 2012 to March 2013

:file:`${PROJECT}/website/data/changes.rst` and :file:`${PROJECT}/adm/website/Readme.rst` are updated. 

.. _saveold:

Save old version
----------------

.. parsed-literal::

   $ datemodif="20130725"

   $ export datemodif
   $ cd /usr/lodyc/incas/fplod/tropflux_d/
   $ mkdir obsolete_${datemodif}

.. parsed-literal::

   $ mv Readme.rst obsolete_${datemodif}
   $ mv ./daily obsolete_${datemodif}
   $ mv ./monthly obsolete_${datemodif}

.. _copynew1_20130725:

Copy new version (phase 1)
--------------------------

.. parsed-literal::

   $ mkdir /usr/lodyc/incas/fplod/tropflux_d/to_be_published
   $ newdir='/net/adonis/usr/adonis/varclim/tropflux_v2'
   $ export newdir

Is there enought space ?
   $ du -sh ${newdir}/daily
   12G     /net/adonis/usr/adonis/varclim/tropflux_v2/daily
   $ du -sh ${newdir}/monthly
   396M    /net/adonis/usr/adonis/varclim/tropflux_v2/monthly
   $ df -h /usr/lodyc/incas/fplod/tropflux_d/to_be_published
   Filesystem            Size  Used Avail Use% Mounted on
   fslodyc:/usr/.lodyc   1.6T  1.6T   20G  99% /.autofs/lodyc

20GB vs 12.4 GB = yes

   $ cp ${PROJECT}/adm/website/Readme.rst /usr/lodyc/incas/fplod/tropflux_d/to_be_published/Readme.rst
   $ cp -r ${newdir}/daily /usr/lodyc/incas/fplod/tropflux_d/to_be_published
   $ cp -r ${newdir}/monthly /usr/lodyc/incas/fplod/tropflux_d/to_be_published

.. _compareversion_20130725:

Compare versions
----------------

A set of tools must be run to compare obsolete and new versions

TDB

.. parsed-literal::

   $ ++compare.sh -o obsolete_${datemodif} -n /usr/lodyc/incas/fplod/tropflux_d/to_be_published

   mkdir -p /usr/work/incas/fplod/tropflux_d//compare_update/
   varlist="lhf lwr netflux q2m shf swr t2m tau taux tauy ws"
   period_list="daily monthly"
   lastyear=2012
   for var in $varlist
   do
      for period in ${period_list}
      do 
         if [ "${period}" == "daily" ]
         then 
            suffix=1d
         else
            suffix=1m
         fi
         newfile=/usr/lodyc/incas/fplod/tropflux_d/to_be_published/${period}/${var}_tropflux_${suffix}_1979_march2013.nc
         newfile_oldtimecoverage=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastdate}.nc
         # extract old time coverage in the new file
         nt=$(ncdump -h $oldfile | grep "time = UNLIMITED ;" | sed -e "s@\(time = UNLIMITED ; // (\)\(.*\)\( currently)\)@\2@")
         slabtime=$(printf "time,0,%d" $((${nt} -1 )))
         ncks -d ${slabtime} -O -o ${newfile_oldtimecoverage} ${newfile}
         oldfile=/usr/lodyc/incas/fplod/tropflux_d/obsolete_${datemodif}/${period}/${var}_tropflux_${suffix}_1979_${lastyear}.nc
         oldfiledump=${PROJECT_ID}/compare_update/${var}_tropflux_${suffix}_1979_${lastyear}.nc_ncdump
         cdo diff  ${newfile_oldtimecoverage} ${oldfile}
         read a
         # we must see "0 of $nt records differ"
         ncdump -h ${newfile_oldtimecoverage} > ${newfile_oldtimecoverage}_ncdump
         ncdump -h ${oldfile} > ${oldfiledump}
         diff ${newfile_oldtimecoverage}_ncdump ${oldfiledump}
         # we must see a difference in creation date and coverage
         read a
         rm ${newfile_oldtimecoverage}_ncdump
         rm ${oldfiledump}
         unset suffix
      done
      unset period
      unset period_list
   done
   unset var
   unset varlist
   rm -rf /usr/work/incas/fplod/tropflux_d//compare_update/

   log=${PROJECT_LOG}/compare.log
   script ${PROJECT_LOG}/compare.log
   compare.sh
   exit

Check for results.

First check expected equality for data:

::

   grep "records differ" ${log}
  
If only "0 of" begin all lines, its ok. Otherwise look inside the log file to
locate troubles.

Second, check for differences on global attributes

::

   grep "^>" ${log} | grep creation_date

If nothing appears, you must modified this attributes. see :ref:`change_creation_date.sh` ++


   grep "^>" ${log} | grep time_range

If nothing appears, you must modified this attributes. see :ref:`change_timerange.sh` ++

Clean log file::

   rm ${log}

If modifications has been made, re-run comparaison before next step

Once comparison is done and PI happy,
++ cleanning or modify newdir

.. _copynew2_20130725:

Copy new version (phase 2)
--------------------------

.. parsed-literal::

   $ cd /usr/lodyc/incas/fplod/tropflux_d/
   $ mv ./to_be_published/Readme.rst .
   $ mv ./to_be_published/daily ./
   $ mv ./to_be_published/monthly ./
   $ ln -sf $(pwd)/Readme.rst daily/
   $ ln -sf $(pwd)/Readme.rst monthly/


.. _desinstalldata_20130725:

Remove old version from the website
-----------------------------------

To avoid confusion, old version has to be removed from the website:

.. parsed-literal::

   fplod@sonmac$ ftp www.locean-ipsl.upmc.fr
   ftp> cd |project_id|/data/
   ftp> mdelete daily/*
   ftp> mdelete monthly/*

.. _installdata_20130725:

Installation of the new version
-------------------------------

L'installation se fait ainsi:

.. parsed-literal::

   fplod@sonmac$ cd /Volumes/Lcerbere/tropflux_d/
   fplod@sonmac$ ftp www.locean-ipsl.upmc.fr
   ftp> cd |project_id|/data/
   ftp> put Readme.rst
   ftp> cd daily
   ftp> lcd daily
   ftp> mput \*.nc
   ftp> put ../Readme.rst
   ftp> cd ../monthly
   ftp> lcd ../monthly
   ftp> put ../Readme.rst
   ftp> mput \*.nc


.. note::

   :file:`/Volumes/Lcerbere/tropflux_d/` is equivalent to file:`/usr/lodyc/incas/fplod/tropflux_d/` thanks to sshfs command.

   .. parsed-literal::

      $ diskext="/Volumes/LCerbere"
      $ mkdir ${diskext}
      $ sshfs fplod@cerbere:/usr/lodyc/incas/fplod ${diskext} -odefer_permissions -o sshfs_debug


 et template pour NEW

et build doc et install doc

(see also :ref:`builddoc` to synchronize dataset and documentation).


et svn commit ppur ce fichier et tous ceux mentionnes ici
rm /usr/lodyc/incas/fplod/tropflux_d/to_be_published/