source: CPL/oasis3/trunk/src/mod/oasis3/doc/UG6_Compilation_running.tex @ 1677

\newpage
%

\chapter{Compiling and running with OASIS3}
\label{sec_compilationrunning}

OASIS3 and its TOYCLIM coupled model has been successfully compiled
and run on Fujitsu VPP5000, NEC SX5 and SX6, SGI IRIX64, SGI Origin
3800, Linux Opteron, IBM Power4, and Cray X1.

\section{Compiling OASIS3 and the TOYCLIM coupled model}
\label{subsec_compile}

OASIS3 and the TOYCLIM coupled model use the PRISM standard directory
structure (see also \cite{leg04}) and Standard Compiling Environment (see
also \cite{gay04}). To
compile OASIS3 and toyatm, toyoce and toyche component models, one
should go through the following steps:
\begin{enumerate}
\item Go in the directory {\tt prism/util/compile/frames}.

\item Create the include files for your platform if they do not
already exist in directory \break {\tt
prism/util/compile/frames/include\_<node>} where {\tt <node>} is the
name of the platform. 

\item Generate a compile script for the libraries using the script
Create\_COMP\_libs.frm:

\vspace*{2ex}
\begin{Frame}
  \vspace*{1ex}
  \Unixcmd{Create\_COMP\_libs.frm "" "" "" ""}
\end{Frame}
\vspace*{2ex}

The first parameter can be either {\tt ""} or {\tt "-"} to direct the
standard output to a file or the screen.

The second parameter an be either {\tt ""}, {\tt "-"} or {\tt "+"} to
direct the standard error to a file, the screen or the standard
output.

If the compile scripts shall be created for another platform than the
one where the \break Create\_COMP\_libs.frm script is launched, the third
parameter has to contain the abbreviated node name ``node".

The compile script for the libraries \texttt{COMP\_libs.$<$node$>$}
should then be created in the directory \texttt{prism/util}.
 
\item Generate a compile scrip for OASIS3 and for each of the component
models using the script Create\_COMP\_models.frm:
\vspace*{2ex}
\begin{Frame}
  \vspace*{1ex}
  \Unixcmd{Create\_COMP\_models.frm oasis3 "mp" "" "" "" }\\
  \Unixcmd{Create\_COMP\_models.frm toyoce "mp" "" "" "" "ID" "toyatm toyoce toyche"}\\
  \Unixcmd{Create\_COMP\_models.frm toyatm "mp" "" "" "" "ID" "toyatm toyoce toyche"}\\
  \Unixcmd{Create\_COMP\_models.frm toyche "mp" "" "" "" "ID" "toyatm toyoce toyche"}
\end{Frame}
\vspace*{2ex}

The second parameter ``mp'' specifies the message passing used, which
determines how the models are launched (see also section
\ref{subsubsec_Initialisation}).  If the default `MPI2' is chosen, the
string has to be empty (specification of MPI2 results in an error);
otherwise, MPI1 has to be given, or NONE for the interpolator only
mode -see section \ref{subsec_interpolator}.  The OASIS3 executable
will have the string MPI1 or MPI2 appended to its name. The 3 toy
models can also be compiled with either the MPI1 option or the default
MPI2 option (empty string).

The third parameter can be either {\tt ""} or {\tt "-"} to direct the
standard output to a file or the screen.

The fourth parameter an be either {\tt ""}, {\tt "-"} or {\tt "+"} to
direct the standard error to a file, the screen or the standard
output.

If the compile scripts shall be created for another platform than the
one where the \break Create\_COMP\_models.frm script is launched, the fifth
parameter has to contain the abbreviated node name ``node".

The sixth parameter ``ID'' is version acronym for differentiation of
executables (not relevant for OASIS3 and TOYCLIM toy models).

Finally the last parameter gives the name of all the component models
in the coupled constellation. This list is not relevant for OASIS3,
but it has to be given for the toymodels. The specified partner models
are checked against allowed partners and no default is set.

The scripts to compile OASIS3 and the 3 toy coupled models,
\texttt{COMP\_oasis3\_<mp>.<node>}
\texttt{COMP\_toyatm\_<ID>.<node>}, \texttt{COMP\_toyoce\_<ID>.<node>},
\texttt{COMP\_toyche\_<ID>.<node>} should then be created, respectively in
directories \texttt{prism/src/mod/oasis3}, \texttt{ /toyatm},
\break \texttt{ /toyoce}, \texttt{ /toyche}. 

\item The compilation scripts created can now be used to compile
OASIS3 and the 3 toy models. All four compile scripts have then to be
launched explicitely by the user in their respective directory. 
\vspace*{2ex}
\begin{Frame}
  \vspace*{1ex}
  \Unixcmd{COMP\_oasis3\_<mp>.<node>}\\
  \Unixcmd{COMP\_toyatm\_<ID>.<node>}\\
  \Unixcmd{COMP\_toyoce\_<ID>.<node>}\\
  \Unixcmd{COMP\_toyche\_<ID>.<node>}
\end{Frame}
\vspace*{2ex}

The scripts compile the models with the MPI library specified during
their generation.  The script that triggers the update of the
libraries, \texttt{COMP\_libs.$<$node$>$}, is automatically called by
the model compilation scripts for the librairies they need. Libraries
needed by OASIS3 are anaisg, anaism, scrip, fscint, and clim for MPI1
and MPI2 mode (clim is not compiled in NONE mode). The toy
models need psmile and mpp\_io.
 
\item The result should be executables {\tt oasis3.<mp>.x}, {\tt
toyatm.<mp>.x}, {\tt toyoce.<mp>.x}, and {\tt toyche.<mp>.x} in the
{\tt \$BLDROOT/bin} directory defined by the compile scripts, where
{\tt <mp>} is either MPI1 or MPI2.

\end{enumerate}

\section{Configuring the TOYCLIM coupled model using OASIS3}
\label{subsec_configuring}

The TOYCLIM coupled model performs a coupling between an ``empty" ocean
model, toyoce, an``empty" atmospheric model, toyatm, and an ``empty"
atmospheric chemistry model, toyche. There is no real physics or
dynamics within the models. However, the coupling fields have a
realistic size, the operations performed within OASIS3 on the coupling
fields are realistic, and the coupling using the PRISM System
model interface (PSMILe) is also realistic.  The TOYCLIM atmosphere
and chemistry models have the same grid and the same parallel
partitioning on 3 processes. There is no need of interpolation and
the coupling fields are directly exchanged between these two models
without going through OASIS3 interpolation process. More details on
the TOYCLIM coupled model can be found in \cite{val04b}.

The TOYCLIM coupled model uses PRISM standard running environment
(SRE). To run it, one has to go through the following
steps: 

\begin{enumerate}
\item Go to the directory {\tt prism/util/running/frames}
\item Create the include files for your platform if they do not
already exist in directory {\tt
prism/util\break/running/frames/include\_<node>} where {\tt <node>} is
the name of the platform.
\item Run the script Create\_TASKS.frm to generate a setup file for
your TOYCLIM experiment:

\begin{Frame}
 \begin{Indent}
  \vspace*{1ex}
  \Unixcmd{Create\_TASKS.frm toyclim <expid>}
 \end{Indent}
\end{Frame}
\vspace*{2ex}

where {\tt <expid>} is your experiment name.

\item To change the configuration of your experiment, modify the
values of the configurable entries in the setup file {\tt
prism/util/running/frames/setup/setup\_toyclim\_<expid>}, which
contains default values for these entries. Some of these configurable
entries directly enter the OASIS3 configuration file {\it namcouple},
other affect the running script only\footnote{Additional {\it
namcouple} entries are also configurable by editing directly the
namcouple base file. Refer to chapter \ref{sec_namcouple} for more
details.}.  The {\it namcouple} file is created from the {\it
namcouple} base file (see \texttt{namcouple\_toyclim} in
\texttt{prism/util/running\break/adjunct\_files/oasis3}) by replacing the
configurable entries (which begin with "\#") by the value defined in
the setup file.  The {\it namcouple} will be read by OASIS3 at
runtime. The variables that can be defined in the set-up file and
correspond to configurable {\it namcouple} entries are the following:
 
\begin{enumerate}

\item{\em jobname}: Experiment identifier; it is composed of three
 digits; ({\em \#Jobname}).
 
\item{\em nlogprt}: Integer controlling the amount of information
 written to the OASIS output file {\em cplout}. 0: minimum output, 1:
 medium output, 2: maximum output; ({\em \#Nlogprt}).

\item{\em extrapwr}: Flag to provoke the calculation of weights and
 addresses for nearest neighbour extrapolation (EXTRAP/NINENN) within
 OASIS3 (1) or to read them from file (0); ({\em \#Extrapwr}).

\item{\em stat\_field{\bf xx}}, where {\bf xx} is the number of the
field in the namcouple: The status of the {\bf xx} coupling/IO field
can be either `EXPOUT' or `EXPORTED', except for $Field_3$ for which
it is `INPUT', for $Field_5$ for which it is `OUTPUT' and for
$Field_{10}$ and $Field_{11}$ for which it is either `IGNOUT' or
`IGNORED' (see section \ref{subsec_namcouplesecond}); ({\em
\#Stat\_field{\bf xx}}).

\item{\em dtF{\bf xx}}, where {\bf xx} is the number of the field in
the namcouple: The coupling or I/O period of the {\bf xx} coupling/IO field,
which must be a multiple of 14400, except for $Field_3$ for which it
must be a multiple of 43200, and for $Field_5$, $Field_{10}$ and
$Field_{11}$ for which it must be a multiple of 3600; ({\em \#Dt{\bf
xx}}).

\item {\em iniyear}, {\em inimonth} and {\em iniday}: the initial date
of the experiment, respectively as {\tt YYYY}, {\tt MM} and {\tt
DD}({\em \#Inidate}).

\item {\em message\_passing}: Message passing method, either MPI2 or
  MPI1 (for more details, see section \ref{subsubsec_Initialisation});
  (in {\em \#Channel})

\item {\em bsend}: either `yes' or `no', indicates whether the
 buffered {\tt MPI\_BSend} or the standard blocking send {\tt
 MPI\_Send} will be used to send the coupling fields (for more
 details, see section \ref{subsec_namcouplefirst}) (in {\em \#Channel})

\end{enumerate}

The variables that can be additionally defined in the setup file but
do not correspond to any
configurable {\it namcouple} entries are the following:
 
\begin{enumerate}

\item {\em ncplvers}: the namcouple version. Put {\tt ""} to use the
  {\it namcouple} base file completed with the values defined in the
  setup file. To use another {\it namcouple}, a particular value has
  to be given to {\em ncplvers}, and a {\it namcouple} named
  {\tt namcouple\_toyclim\_$<$ncplvers$>$} has to be available in
  \texttt{prism/util/running/adjunct\_files/oasis3}.

\item{\em gridswr}: either `0' if you want the models to use the grid
  description files if they exist, or `1' if you want the models to
  unconditionally (re)generate those grid description files (for more
 details, see section \ref{subsubsec_griddef}).

\end{enumerate}

\item Type {\tt `Create\_TASKS.frm toyclim <expid>'} a second
time. The script will check the parameters you specified in
{\tt setup\_toyclim\_<expid>}. If a parameter is not supported or a
combination of parameters does not make sense Create\_TASKS.frm writes
an error message and stops.
\item Correct the experimental setup file if necessary and
run Create\_TASKS.frm again until the setup check is passed
successfully.

\end{enumerate}

Once the setup is done, all appropriate files and the script to start
the experiment are available in the
directory {\tt $<$home$>$/$<$expid$>$}, where {\tt $<$home$>$} and
{\tt $<$expid$>$} are
defined in the setup file.

\section{Running the TOYCLIM coupled model using OASIS3}
\label{subsec_running}

After the setup and configuration phase, the experiment is ready to be
started with the running script {\tt RUN\_toyclim\_<expid>} in
directory {\tt $<$work$>$/$<$expid$>$/scripts}, where {\tt $<$work$>$} and
{\tt $<$expid$>$} are
as defined in the setup file {\tt setup\_toyclim\_<expid>}.

To run a TOYCLIM experiment, one has to go through the following
steps:

\begin{enumerate} 
%\item The file {\tt
%  toyclim/input\_toyclim\_standard\_\-standard\_prism\_2-1.tar.gz} has
%  to be copied from the directory {\tt prism/data} to the directory
%{\tt <archive\_in>/toyclim} on the \break {\tt <archiving\_host>}.  The variables
%{\tt <archive\_in>} and {\tt <archiving\_host>} are specified in the setup file .
\item Login on the compute server (if different from the compile server).
\item Change to the directory {\tt <work>/<expid>/scripts}. 
\item Submit the runscript {\tt RUN\_toyclim\_<expid>}.
\end{enumerate}

Running TOYCLIM with the default parameters will result in a 30-day
experiment executed as five 6-day runs. The final results obtained the
directory {\tt <work>/<expid>/work} should match the ones in {\tt
prism/data/toyclim/outdata}. Intermediate results are also saved in
different sub-directories :
\begin{itemize} 
 \item the OASIS3 log files of each
run in {\tt <data>/<expid>/log } or {\tt  <archive>/<expid>/log }
 \item the output netCDF files containing the {\tt EXPOUT} fields in {\tt
<data>/<expid>/outdata/oasis3 } or {\tt
 <archive>/<expid>/outdata/oasis3 }
\item the coupling restart files in {\tt  <data>/<expid>/restart } or
  {\tt  <archive>/<expid>/restart}.
\end{itemize}
where {\tt <data>} and {\tt <archive>} are
defined in the runscript {\tt RUN\_toyclim\_<expid>}.

\section{Running the TOYCLIM coupled model manually}
\label{subsec_runningmanually}

If you want to run the TOYCLIM manually, you need to copy the
following files in your working directory:
\begin{itemize}
\item OASIS3, toyatm, toyoce, toyche executables
\item OASIS3 grid, restart, and auxiliary files: grids.nc, masks.nc,
 areas.nc, SOALBEDO.nc, flda1.nc, flda2.nc, flda3.nc,
 flda4.nc, fldo1.nc, at31topa, runoff31, which are in
{\tt  prism/data/toyclim\break/input\_toyclim\_standard\_standard\_prism\_2-2.tar.gz}
\item OASIS3 adjunct files `cf\_name\_table.txt' and `namcouple' which can be
 found under names \break `cf\_name\_table.txt' and
 `namcouple\_toyclim\_use' in {\tt prism/util/running/adjunct\_files\break/oasis3}
\end{itemize}
Then you need to start the coupled model, setting up MPI parameters
properly. For examples, refer to files {\tt config\_site\_<node>.h} and
{\tt launching\_toyclim\_<node>.h} in {\tt
prism/util/running\break/frames/include\_<node>} (where {\tt <node>} is the
name of the platform).  

\section{Running OASIS3 in interpolator-only mode}

To run OASIS3 in interpolator-only mode, one has to go through the
following steps (the following specific files can be
retrieved from CERFACS CVS only):
\begin{itemize}
\item Compile OASIS3 following the procedure described in section
\ref{subsec_compile} specifying {\tt NONE} as {\tt <mp>}.
\item Compile the programs that will calculate the
interpolation error fields, i.e. gen\_error.f90 and \break gen\_error\_vector.f90 in directory {\tt prism/util/running/testinterp/error}. 
\item Execute the
running script {\tt prism/util/running/testinterp/sc\_run\_testinterp}. 
\item The results obtained after running the TOYCLIM with the defaults
parameters should match the ones in {\tt prism/data/testinterp/outdata}.
\end{itemize}
For more details, please read the {\tt
prism/util/running/testinterp/README\_testinterp}.