Changeset 4063

Timestamp:

2013-10-15T18:35:30+02:00 (11 years ago)

Author:

djlea

Message:

Include offline observation operator documentation.

File:

• branches/2013/dev_r3987_UKMO4_OBS/DOC/TexFiles/Chapters/Chap_OBS.tex (modified) (3 diffs)

branches/2013/dev_r3987_UKMO4_OBS/DOC/TexFiles/Chapters/Chap_OBS.tex

@@ -5 +5 @@
 \label{OBS}
 
-Authors: D. Lea, M. Martin, K. Mogensen, A. Vidard, A. Weaver...   % do we keep that ?
+Authors: D. Lea, M. Martin, K. Mogensen, A. Vidard, A. Weaver, A. Ryan, ...   % do we keep that ?
 
 \minitoc
@@ -42 +42 @@
 where to obtain data and how to setup the namelist. Section~\ref{OBS_details} introduces some
 more technical details of the different observation types used and also shows a more complete
-namelist. Section~\ref{OBS_theory} introduces some of the theoretical aspects of the
-observation operator including interpolation methods and running on multiple processors.
-Section~\ref{OBS_obsutils} introduces some utilities to help working with the files produced
-by the OBS code.
+namelist. Section~\ref{OBS_theory} introduces some of the theoretical aspects of the observation
+operator including interpolation methods and running on multiple processors.
+Section~\ref{OBS_ooo} describes the offline observation operator code.
+Section~\ref{OBS_obsutils} introduces some utilities to help working with the files
+produced by the OBS code.
 
 % ================================================================
@@ -786 +787 @@
 \newpage
 
+% ================================================================
+% Offline observation operator documentation
+% ================================================================
+
+%\usepackage{framed}
+
+\section{Offline observation operator}
+\label{OBS_ooo}
+
+\subsection{Concept}
+
+The obs oper maps model variables to observation space. It is possible to apply this mapping
+without running the model. The software which performs this functionality is known as the
+\textbf{offline obs oper}. The obs oper is divided into three stages. An initialisation phase,
+an interpolation phase and an output phase. The implementation of which is outlined in the
+previous sections. During the interpolation phase the offline obs oper populates the model
+arrays by reading saved model fields from disk.
+
+There are two ways of exploiting this offline capacity. The first is to mimic the behaviour of
+the online system by supplying model fields at regular intervals between the start and the end
+of the run. This approach results in a single model counterpart per observation. This kind of
+usage produces feedback files the same file format as the online obs oper. 
+The second is to take advantage of the offline setting in which multiple model counterparts can
+be calculated per observation. In this case it is possible to consider all forecasts verifying
+at the same time. By forecast, I mean any method which produces an estimate of physical reality
+which is not an observed value. In the case of class 4 files this means forecasts, analyses, persisted
+analyses and climatological values verifying at the same time. Although the class 4 file format
+doesn't account for multiple ensemble members or multiple experiments per observation, it is possible
+to include these components in the same or multiple files.
+
+%--------------------------------------------------------------------------------------------------------
+% offline_oper.exe 
+%--------------------------------------------------------------------------------------------------------
+
+\subsection{Using the offline observation operator}
+
+\subsubsection{Building}
+
+In addition to the code required to build nemo.exe the offline obs oper requires the inclusion
+of the \emph{OOO\_SRC directory} and \emph{offline\_oper.exe} to be specified as a target.
+Depending on your build system this can be achieved in several ways. But for users of
+the \textbf{makenemo} script, a flag has been added which adds OOO\_SRC to the 
+\textbf{config.txt} and which also adds a secondary target to the \textbf{bld.cfg}.
+
+%--------------------------------------------------------------------------------------------------------
+% Running 
+%--------------------------------------------------------------------------------------------------------
+\subsubsection{Running}
+
+The simplest way to use the executable is to edit and append the \textbf{ooo.nml} namelist to
+a full NEMO namelist and then to run the executable as if it were nemo.exe. 
+
+%\subsubsection{Scripting}
+%
+%Not yet implemented.
+
+%--------------------------------------------------------------------------------------------------------
+% Configuration section
+%--------------------------------------------------------------------------------------------------------
+\subsection{Configuring the offline observation operator}
+The observation files and settings understood by \textbf{namobs} have been outlined in the online
+obs oper section. In addition there are two further namelists wich control the operation of the offline
+obs oper. \textbf{namoff} which controls the input model fields and \textbf{namcl4} which controls the
+production of class 4 files. 
+
+\subsubsection{Single field}
+
+In offline mode model arrays are populated at appropriate time steps via input files.
+At present, \textbf{tsn} and \textbf{sshn} are populated by the default read routines. 
+These routines will be expanded upon in future versions to allow the specification of any
+model variable. As such, input files must be global versions of the model domain with
+\textbf{votemper}, \textbf{vosaline} and optionally \textbf{sshn} present.
+
+For each field read there must be an entry in the \textbf{namoff} namelist specifying the
+name of the file to read and the index along the \emph{time\_counter}. For example, to
+read the second time counter from a single file the namelist would be.
+
+\begin{alltt}
+\tiny
+\begin{verbatim} 
+!----------------------------------------------------------------------
+!       namoff Offline obs_oper namelist
+!----------------------------------------------------------------------
+!   off_files    specifies the files containing the model counterpart
+!   nn_off_idx   specifies the time_counter index within the model file
+&namoff
+   off_files = "foo.nc"
+   nn_off_idx = 2
+/
+\end{verbatim} 
+\end{alltt}
+
+\subsubsection{Multiple fields per run}
+
+Model field iteration is controlled via \textbf{nn\_off\_freq} which specifies
+the number of model steps at which the next field gets read. For example, if
+12 hourly fields are to be interpolated in a setup where 288 steps equals 24 hours.
+
+\begin{alltt}
+\tiny
+\begin{verbatim} 
+!----------------------------------------------------------------------
+!       namoff Offline obs_oper namelist
+!----------------------------------------------------------------------
+!   off_files    specifies the files containing the model counterpart
+!   nn_off_idx   specifies the time_counter index within the model file
+!   nn_off_freq  specifies number of time steps between read operations
+&namoff
+   off_files = "foo.nc" "foo.nc"
+   nn_off_idx = 1 2
+   nn_off_freq = 144
+/
+\end{verbatim} 
+\end{alltt}
+
+The above namelist will result in feedback files whose first 12 hours contain
+the first field of foo.nc and the second 12 hours contain the second field.
+
+%\begin{framed}
+\textbf{Note} Missing files can be denoted as "nofile".
+%\end{framed}
+
+It is easy to see how a collection of fields taken fron a number of files 
+at different indices can be combined at a particular frequency in time to 
+generate a pseudo model evolution. As long as all that is needed is a single
+model counterpart at a regular interval then namoff is all that needs to
+be edited. However, a far more interesting approach can be taken in which
+multiple forecasts, analyses, persisted analyses and climatologies are
+considered against the same set of observations. For this a slightly more
+complicated approach is needed. It is referred to as \emph{Class 4} since
+it is the fourth metric defined by the GODAE intercomparison project.
+
+%--------------------------------------------------------------------------------------------------------
+% Class 4 file section
+%--------------------------------------------------------------------------------------------------------
+\subsubsection{Multiple model counterparts per observation a.k.a Class 4}
+
+A generalisation of feedback files to allow multiple model components per observation. For a single
+observation, as well as previous forecasts verifying at the same time there are also analyses, persisted
+analyses and climatologies. 
+
+
+The above namelist performs two basic functions. It organises the fields
+given in \textbf{namoff} into groups so that observations can be matched
+up multiple times. It also controls the metadata and the output variable
+of the class 4 file when a write routine is called.
+
+%\begin{framed}
+\textbf{Note: ln\_cl4} must be set to \emph{.TRUE.} in \textbf{namobs} 
+to use class 4 outputs.
+%\end{framed}
+
+\subsubsection{Class 4 naming convention}
+
+The standard class 4 file naming convention is as follows.
+
+\noindent
+\linebreak
+\textbf{\$\{prefix\}\_\$\{yyyymmdd\}\_\$\{sys\}\_\$\{cfg\}\_\$\{vn\}\_\$\{kind\}\_\$\{nproc\}.nc}
+
+\noindent
+\linebreak
+Much of the namelist is devoted to specifying this convention. The
+following namelist settings control the elements of the output
+file names. Each should be specified as a single string of character data.
+
+\begin{description}
+\item[cl4\_prefix]
+Prefix for class 4 files e.g. class4
+\item[cl4\_date]
+YYYYMMDD validity date
+\item[cl4\_sys]
+The name of the class 4 model system e.g. FOAM
+\item[cl4\_cfg]
+The name of the class 4 model configuration e.g. orca025
+\item[cl4\_vn]
+The name of the class 4 model version e.g. 12.0
+\end{description}
+
+\noindent
+The kind is specified by the observation type internally to the obs oper. The processor
+number is specified internally in NEMO. 
+
+\subsubsection{Class 4 file global attributes}
+
+Global attributes necessary to fulfill the class 4 file definition. These
+are also useful pieces of information when collaborating with external
+partners.
+
+\begin{description}
+\item[cl4\_contact]
+Contact email for class 4 files.
+\item[cl4\_inst]
+The name of the producers institution.
+\item[cl4\_cfg]
+The name of the class 4 model configuration e.g. orca025
+\item[cl4\_vn]
+The name of the class 4 model version e.g. 12.0
+\end{description}
+
+\noindent
+The obs\_type,
+creation date and validity time are specified internally to the obs oper.
+
+\subsubsection{Class 4 model counterpart configuration}
+
+As seen previously it is possible to perform a single sweep of the
+obs oper and specify a collection of model fields equally spaced 
+along that sweep. In the class 4 case the single sweep is replaced
+with multiple sweeps and a certain ammount of book keeping is
+needed to ensure each model counterpart makes its way to the 
+correct piece of memory in the output files.
+
+\noindent
+\linebreak
+In terms of book keeping, the offline obs oper needs to know how many
+full sweeps need to be performed. This is specified via the 
+\textbf{cl4\_match\_len} variable and is the total number of model
+counterparts per observation. For example, a 3 forecasts plus 3 persistence
+fields plus an analysis field would be 7 counterparts per observation.
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+   cl4_match_len = 7
+\end{verbatim}
+\end{alltt}
+
+Then to correctly allocate a class 4 file the forecast axis must be defined. This
+is controlled via \textbf{cl4\_fcst\_len}, which in out above example would be 3.
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+   cl4_fcst_len = 3
+\end{verbatim}
+\end{alltt}
+
+Then for each model field it is necessary to designate what class 4 variable and
+index along the forecast dimension the model counterpart should be stored in the
+output file. As well as a value for that lead time in hours, this will be useful
+when interpreting the data afterwards. 
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+   cl4_vars = "forecast" "forecast" "forecast" "persistence" "persistence"
+              "persistence" "best_estimate"
+   cl4_fcst_idx = 1 2 3 1 2 3 1
+   cl4_leadtime = 12 36 60 
+\end{verbatim}
+\end{alltt}
+
+In terms of files and indices of fields inside each file the class 4 approach
+makes use of the \textbf{namoff} namelist. If our fields are in separate files
+with a single field per file our example inputs will be specified.
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+   off_files = "F.1.nc" "F.2.nc" "F.3.nc" "P.1.nc" "P.2.nc" "P.3.nc" "A.1.nc"
+   nn_off_idx = 1 1 1 1 1 1 1
+\end{verbatim}
+\end{alltt}
+
+When we combine all of the naming conventions, global attributes and i/o instructions
+the class 4 namelist becomes.
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+!----------------------------------------------------------------------
+!       namoff Offline obs_oper namelist
+!----------------------------------------------------------------------
+!   off_files    specifies the files containing the model counterpart
+!   nn_off_idx   specifies the time_counter index within the model file
+!   nn_off_freq  specifies number of time steps between read operations
+&namoff
+   off_files = "F.1.nc" "F.2.nc" "F.3.nc" "P.1.nc" "P.2.nc" "P.3.nc" "A.1.nc"
+   nn_off_idx = 1 1 1 1 1 1 1
+/
+!----------------------------------------------------------------------
+!       namcl4 Offline obs_oper class 4 namelist
+!----------------------------------------------------------------------
+!
+!  Naming convention
+!  -----------------
+!  cl4_prefix    specifies the output file prefix
+!  cl4_date      specifies the output file validity date
+!  cl4_sys       specifies the model counterpart system
+!  cl4_cfg       specifies the model counterpart configuration
+!  cl4_vn        specifies the model counterpart version
+!  cl4_inst      specifies the model counterpart institute
+!  cl4_contact   specifies the file producers contact details
+!
+!  I/O specification
+!  -----------------
+!  cl4_vars      specifies the names of the output file netcdf variable
+!  cl4_fcst_idx  specifies output file forecast index
+!  cl4_fcst_len  specifies forecast axis length
+!  cl4_match_len specifies number of unique matches per observation
+!  cl4_leadtime  specifies the forecast axis lead time 
+!
+&namcl4
+   cl4_match_len = 7
+   cl4_fcst_len = 3
+   cl4_fcst_idx = 1 2 3 1 2 3 1
+   cl4_vars = "forecast" "forecast" "forecast" "persistence" "persistence"
+              "persistence" "best_estimate"
+   cl4_leadtime = 12 36 60
+   cl4_prefix = "class4"
+   cl4_date = "20130101"
+   cl4_vn = "12.0"
+   cl4_sys = "FOAM"
+   cl4_cfg = "AMM7"
+   cl4_contact = "example@example.com"
+   cl4_inst = "UK Met Office"
+/
+\end{verbatim}
+\end{alltt}
+
+\subsubsection{Climatology interpolation}
+
+The climatological counterpart is generated at the start of the run by restarting 
+the model from climatology through appropriate use of \textbf{namtsd}. To override
+the offline observation operator read routine and to take advantage of the restart
+settings, specify the first entry in \textbf{cl4\_vars} as "climatology". This will then
+pipe the restart from climatology into the output class 4 file. As in every other
+class 4 matchup the input file, input index and output index must be specified.
+These can be replaced with dummy data since they are not used but they must be
+present to cycle through the matchups correctly. 
+
+\subsection{Advanced usage}
+
+In certain cases it may be desirable to include both multiple model fields per
+observation window with multiple match ups per observation. This can be achieved
+by specifying \textbf{nn\_off\_freq} as well as the class 4 settings. Care must
+be taken in generating the off\_files list such that the files are arranged into
+consecutive blocks of single match ups. For example, 2 forecast fields 
+of 12 hourly data would result in 4 separate read operations but only 2 write
+operations, 1 per forecast.
+
+\begin{alltt}
+\tiny
+\begin{verbatim}
+   off_files = "F1.nc" "F1.nc" "F2.nc" "F2.nc"
+...
+   cl4_fcst_idx = 1 2
+\end{verbatim}
+\end{alltt}
+
+The above notation reveals the internal split between match up iterators and file
+iterators. This technique has not been used before so experimentation is needed
+before results can be trusted.
+
+
+
+
+\newpage
+
 \section{Observation Utilities}
 \label{OBS_obsutils}