source: trunk/NEMO/DOC/BETA/Chapters/Chap_LBC.tex @ 705

% ================================================================
% Chapter Ñ Lateral Boundary Condition (LBC) 
% ================================================================
\chapter{Lateral Boundary Condition (LBC) }
\label{LBC}
\minitoc

%gm% add here introduction to this chapter

% ================================================================
% Boundary Condition at the Coast
% ================================================================
\section{Boundary Condition at the Coast (\np{shlat})}
\label{LBC_coast}
%--------------------------------------------nam_lbc-------------------------------------------------------
\namdisplay{nam_lbc} 
%--------------------------------------------------------------------------------------------------------------

%The lateral ocean boundary condition continuous to coastlines are Neumann conditions for heat and salt (no flux across boundaries) and Dirichlet conditions for momentum (from free-slip to "strong" no-slip). They are handled automatically by the mask system (see \S\ref{DOM_msk}). 

%OPA works with land and topography grid points in the computational domain due to the presence of continents or islands,and to the use of full or partial step representation of bottom topography. The computation is performed over the whole domain, i.e. we did not try to restrict the computation to ocean point only areas. This choice has two motivations. First, working on ocean only grid point overload the code and harms the code readability. Second,, and more importantly, it drastically reduce the vector portion of the computation, leading to a dramatic increase of CPU time requirement on vector computers. The process of defining which areas are to be masked is described in \S\ref{DOM_msk}, while this section describes how the masking affects the computation of the various terms of the equations with respect to the boundary condition at solid walls.

The discrete representation of a domain with complex boundaries (coastlines and bottom topography) leads to arrays that include large portions where a computation is not required as the model variables remain at zero. Nevertheless, vectorial supercomputers are far more efficient when computing over a whole array, and the readability of a code is greatly improved when boundary conditions are applied in an automatic way rather than by a specific computation before or after each do loop. An efficient way to work over the whole domain while specifying the boundary conditions is to use the multiplication by mask arrays in the computation. A mask array is a matrix 
which elements are $1 $in the ocean domain and $0$ elsewhere. A simple multiplication of a variable by its own mask ensures that it will remain zero over land areas. Since most of the boundary conditions consist of a zero flux across the solid boundaries, they can be simply settled by 
multiplying variables by the right mask arrays, i.e. the mask array of the grid point where the flux is evaluated. For example, the heat flux in the \textbf{i}-direction is evaluated at $u$-points. Evaluating this quantity as,

\begin{equation} \label{Eq_lbc_aaaa}
\frac{A^{lT} }{e_1 }\frac{\partial T}{\partial i}\equiv \frac{A_u^{lT} 
}{e_{1u} }\delta _{i+1 / 2} \left[ T \right]\;\;mask_u 
\end{equation}

%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!t] \label{Fig_LBC_uv}  \begin{center}
\includegraphics[width=0.90\textwidth]{./Figures/Fig_LBC_uv.pdf}
\caption {Lateral boundary (thick line) at T-level. The velocity normal to the boundary is set to zero.}
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>

where mask$_{u}$ is the mask array at $u$-point, ensures that the heat flux is 
zero inside land and at the boundaries as mask$_{u}$ is zero at solid 
boundaries defined at $u$-points in this case (normal velocity $u$ remains zero at 
the coast) (Fig.~\ref{Fig_LBC_uv}). 

On momentum the situation is a bit more complex as two boundary conditions must be provided along the coast (one on the normal velocity and the other on the tangential velocity). The boundary of the ocean in C-grid is defined by the velocity-faces. For example, at a given $T$-level, the lateral boundary (coastline or intersection with the bottom topography) is made of segments 
joining $f$-points, and normal velocity points are located between two $f-$points (Fig.~\ref{Fig_LBC_uv}). The boundary condition on the normal velocity (no flux through solid boundaries) can thus be easily settled by the mask system. The boundary condition on the tangential velocity requires a more specific treatment. It influences the relative vorticity and momentum diffusive trends, and is only required to compute the vorticity at the coast. Four different type of the lateral boundary condition are available, controlled by the value of \np{shlat} namelist parameter (which is equal to the value of the mask$_{f}$ array along the coastline):

%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!p] \label{Fig_LBC_shlat}  \begin{center}
\includegraphics[width=0.90\textwidth]{./Figures/Fig_LBC_shlat.pdf}
\caption {lateral boundary condition (a) free-slip ($shlat=0$) ; (b) no-slip ($shlat=2$) ; (c) "partial" free-slip ($0<shlat<2$) and (d) "strong" no-slip ($2<shlat$). Implied "ghost" velocity inside land area is display in grey. }
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>

\begin{description}

\item[free-slip boundary condition (\np{shlat}=0): ]  the tangential velocity at the coastline is equal to the offshore velocity, $i.e.$ the normal derivative of the tangential velocity is zero at the coast, so the vorticity: mask$_{f}$ array is set to zero inside the land and just at the coast (Fig.~\ref{Fig_LBC_shlat}-a).

\item[no-slip boundary condition (\np{shlat}=2): ] the tangential velocity vanishes at the coastline. Assuming that the tangential velocity decreases linearly from the closest ocean velocity grid point to the coastline, the normal derivative is evaluated as if the closest land velocity gridpoint were of the same magnitude as the closest ocean velocity gridpoint but in the opposite direction (Fig.~\ref{Fig_LBC_shlat}-b). Therefore, the vorticity along the coastlines is given by: 
\begin{equation*}
\zeta \equiv 2 \left(\delta_{i+1/2} \left[e_{2v} v \right] - \delta_{j+1/2} \left[e_{1u} u \right] \right) / \left(e_{1f} e_{2f} \right) \ ,
\end{equation*}
where $u$ and $v$ are masked fields. Setting the mask$_{f}$ array to $2$ along the coastline allows to provide a vorticity field computed with the no-slip boundary condition simply by multiplying it by the mask$_{f}$ :
\begin{equation} \label{Eq_lbc_bbbb}
\zeta \equiv \frac{1}{e_{1f} {\kern 1pt}e_{2f} }\left( {\delta _{i+1/2} 
\left[ {e_{2v} \,v} \right]-\delta _{j+1/2} \left[ {e_{1u} \,u} \right]} 
\right)\;\mbox{mask}_f 
\end{equation}

\item["partial" free-slip boundary condition (0$<$\np{shlat}$<$2): ] the tangential velocity at the coastline is smaller than the offshore velocity, $i.e.$ there is a lateral friction but not strong enough to vanish the tangential velocity at the coast (Fig.~\ref{Fig_LBC_shlat}-c). This can be settled by providing a value of mask$_{f}$ strictly inbetween $0$ and $2$.

\item["strong" no-slip boundary condition (2$<$\np{shlat}): ] the viscous boundary layer is assumed to be smaller than half the grid size (Fig.~\ref{Fig_LBC_shlat}-d). The friction is thus larger than in the no-slip case.

\end{description}

Note that when the bottom topography is entirely represented by the $s$-coordinates (pure $s$-coordinate), the lateral boundary condition on momentum tangential velocity is of much  little importance as it is only applied next to the coast where the minimum water depth can be quite shallow.

The alternative numerical implementation of the no-slip boundary conditions for an arbitrary coast line of \citet{Shchepetkin1996} is also available through the activation of \key{noslip\_accurate}. It is based on a fourth order evaluation of the shear at the coast which, in turn, allows a true second order scheme in the interior of the domain ($i.e.$ the numerical boundary scheme simulates the truncation error of the numerical scheme used in the interior of the domain). \citet{Shchepetkin1996} found that such a technique considerably improves the quality of the numerical solution. In \NEMO, the improvement have not been found so spectacular in the half-degree global ocean (ORCA05), but significant reduction of numerically induced coast upwellings were found in eddy resolving simulation of the Alboran Sea \citep{OlivierPh2001}. Nevertheless, as no-slip boundary condition is not recommended in eddy permitting or resolving simulation \citep{Penduff2007}, the use of this option is not recommended.

In practice, the no-slip accurate option changes the way the curl is evaluated at the coast (see \mdl{divcur} module), and requires to qualify the nature of coastline grid point (convex or concave corners, straight north-south or east-west coast) which is performed in \mdl{domask} module, \rou{dom\_msk\_nsa} routine.

% ================================================================
% Boundary Condition around the Model Domain
% ================================================================
\section{Model Domain Boundary Condition (\jp{jperio})}
\label{LBC_jperio}

At the model domain boundaries several choices are offered: closed, cyclic east-west, south symmetric across the equator, a north-fold, and combination closed-north fold or cyclic-north-fold. The north-fold boundary condition is associated with the 3-pole ORCA mesh. 

% -------------------------------------------------------------------------------------------------------------
%        Closed, cyclic, south symmetric (\jp{jperio} = 0, 1 or 2) 
% -------------------------------------------------------------------------------------------------------------
\subsection{Closed, cyclic, south symmetric (\jp{jperio} = 0, 1 or 2)}
\label{LBC_jperio012}

The choice of closed, cyclic or symmetric model domain boundary condition is made by setting \jp{jperio} to 0, 1 or 2 in \mdl{par\_oce} file. Each time such a boundary condition is needed, it is set by a call to \mdl{lbclnk} routine. The computation of momentum and tracer trends proceed from $i=2$ to $i=jpi-1$ and from $j=2$ to $j=jpj-1$, $i.e.$in the inner model domain. To choose a lateral model boundary condition is to specify the first and last rows and columns of the model variables.

- For closed boundary (\textit{jperio=0}), solid walls are imposed at all model boundaries: 
first and last rows and columns are set to zero.

- For cyclic east-west boundary (\textit{jperio=1}), first and last rows are set to zero 
(closed) while first column is set to the value of the before last column 
and last column to the value of the second one (Fig.~\ref{Fig_LBC_jperio}-a). Whatever flows 
out of the eastern (western) end of the basin enters the western (eastern) 
end. Note that there is neither option for north-south cyclic nor doubly 
cyclic cases.

- For symmetric boundary condition across the equator (\textit{jperio=2}), last rows, and 
first and last columns are set to zero (closed). The row of symmetry is 
chosen to be the $u$- and $T-$points equator line ($j=2$, i.e. at the southern end 
of the domain). For arrays defined at $u-$ or $T-$points, the first row is set to 
the value of the third row while for most of $v$- and $f$-points arrays (v, $\zeta$, 
$j\psi$, but scalar arrays such as eddy coefficients) the first row is set to 
minus the value of the second row (Fig.~\ref{Fig_LBC_jperio}-b). Note that this boundary 
condition is not yet available on massively parallel computer 
(\textbf{key{\_}mpp} defined).

%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!t] \label{Fig_LBC_jperio}  \begin{center}
\includegraphics[width=1.0\textwidth]{./Figures/Fig_LBC_jperio.pdf}
\caption {setting of (a) east-west cyclic  (b) symmetric across the equator boundary conditions.}
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>

% -------------------------------------------------------------------------------------------------------------
%        North fold (\textit{jperio = 3 }to $6)$ 
% -------------------------------------------------------------------------------------------------------------
\subsection{North-fold (\textit{jperio = 3 }to $6)$ }
\label{LBC_north_fold}

The north fold boundary condition have been introduced in order to handle the north boundary of an three-polar ORCA grid. Such a grid has two poles in the northern hemisphere. \colorbox{yellow}{to be completed...}

%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!t] \label{Fig_North_Fold_T}  \begin{center}
\includegraphics[width=0.90\textwidth]{./Figures/Fig_North_Fold_T.pdf}
\caption {North fold boundary with a $T$-point pivot and cyclic east-west boundary condition ($jperio=4$), as used in ORCA 2, 1/4, and 1/12. Pink shaded area corresponds to the inner domain mask (see text). }
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>

% ================================================================
% Exchanged with neighbouring processors 
% ================================================================
\section{Exchanged with neighbouring processors (\mdl{lbclnk}, \mdl{lib\_mpp})}
\label{LBC_mpp}

For massively parallel processing (mpp), a domain decomposition method is used. The basis of the method consists in splitting the large computation domain of a numerical experiment into several smaller domains and solving the set of equations by addressing independent local problems. Each processor has its own local memory and computes the model equation over a subdomain of the whole model domain. The subdomain boundary conditions are specified through communications between processors which are explicitly organized by specific statements (message passing method). 

A big advantage is that the method does not need many modifications of the initial FORTRAN code. For the modeller's point of view, each sub domain running on a processor is identical to the "mono-domain" code. In addition, the programmer manages the communications between subdomains, and the code presents more scalability when the number of processors is increased. The porting of OPA code on a iPSC860 was achieved during Guyon's PhD [Guyon et al. 1994, 1995] in collaboration with CETIIS and ONERA. The implementation in the operational context and the studies of performances on a T3D and T3E Cray computers have been made in collaboration with IDRIS and CNRS. The present implementation is largely inspired from Guyon's work  [Guyon 1995].

   The parallelization strategy is defined by the physical characteristics of the ocean model. Second order finite difference schemes leads to local discrete operators that depend at the very most on one neighbouring point. The only non-local computations concerne the vertical physics (implicit diffusion, 1.5 turbulent closure scheme, ...) (delocalization over the whole water column), and the solving of the elliptic equation associated with the surface pressure gradient computation (delocalization over the whole horizontal domain). Therefore, a pencil strategy is used for the data sub-structuration: the 3D initial domain is laid out on local processor memories following a 2D horizontal topological splitting. Each sub-domain computes its own surface and bottom boundary conditions and has a side wall overlapping interface which stocks lateral boundary conditions for computations in the inner sub-domain. The overlapping area is reduced to one row. After a computation, a communication phase starts: each processor sends to its neighbouring processors the update values of the point corresponding to the overlapping area of its neighbouring sub-domain. The communication is done through message passing. Usually the parallel virtual language, PVM, is used as it is a standard language available on  nearly  all MPP cumputers. More specific languages (i.e. computer dependant languages) can be easily used to speed up the communication, such as SHEM on T3E computer. The data exchanges between processors are required at the very place where lateral domain boundary conditions are set in the mono-domain computation (\S III.10-c): the lbc\_lnk routine which manages such conditions is substituted by mpplnk.F or mpplnk2.F routine when running on MPP computer (\key{mpp\_mpi} defined). It has to be noticed that when using MPP version of the model, the east-west cyclic boundary condition is implicitly done, while the south-symmetric boundary condition option is not available.
 
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!t] \label{Fig_mpp}  \begin{center}
\includegraphics[width=0.90\textwidth]{./Figures/Fig_mpp.pdf}
\caption {Positioning of a sub-domain when massively parallel processing is used. }
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>

   In the standard version of the OPA model, the splitting is regular and arithmetic. the i-axis is divided by \jp{jpni} and the j-axis by \jp{jpnj} for a number of processors \jp{jpnij} most often equal to $jpni \times jpnj$ (model parameters set in \mdl{par\_oce}). Each processor is independent and without message passing or synchronous process, programs run alone and access just at its own local memory. For this reason, the main model dimensions are now the local dimensions of the subdomain (pencil) that are noted \jp{jpi}, \jp{jpj}, \jp{jpk}. These dimensions include the internal domain and the overlapping rows. The number of overlapping rows is usually set to one (\jp{jpreci}=1, in \mdl{par\_oce}). The whole domain dimensions are named \jp{jpiglo}, \jp{jpjglo} and \jp{jpk}. The relationship between the whole domain and a sub-domain is:
\begin{eqnarray} 
      jpi & = & ( jpiglo-2*jpreci + (jpni-1) ) / jpni + 2*jpreci  \nonumber \\
      jpj & = & ( jpjglo-2*jprecj + (jpnj-1) ) / jpnj + 2*jprecj  \label{Eq_lbc_jpi}
\end{eqnarray}
where \jp{jpni}, \jp{jpnj} are the number of processors following the i- and j-axis.

\colorbox{yellow}{Figure IV.3: example of a domain splitting with 9 processors and no east-west cyclic boundary conditions.}

   One defines also variables nldi and nlei which correspond to the internal domain bounds, and the variables nimpp and njmpp which are the position of the (1,1) grid-point in the global domain. An element of $T_{l}$, a local array (subdomain) corresponds to an element of $T_{g}$, a global array (whole domain) by the relationship: 
\begin{equation} \label{Eq_lbc_nimpp}
T_{g} (i+nimpp-1,j+njmpp-1,k) = T_{l} (i,j,k),
\end{equation}
with  $1 \leq i \leq jpi$, $1  \leq j \leq jpj $ , and  $1  \leq k \leq jpk$.

   Processors are numbered from 0 to $jpnij-1$, the number is saved in the variable nproc. In the standard version, a processor has no more than four neighbouring processors named nono (for north), noea (east), noso (south) and nowe (west) and two variables, nbondi and nbondj, indicate the situation of the processor \colorbox{yellow}{(see Fig.IV.3)}:
\begin{itemize}
\item       nbondi = -1    an east neighbour, no west processor,
\item       nbondi =  0 an east neighbour, a west neighbour,
\item       nbondi =  1    no east processor, a west neighbour,
\item       nbondi =  2    no splitting following the i-axis.
\end{itemize}
   During the simulation, processors exchange data with their neighbours. If there is effectively a neighbour, the processor receives variables from this processor on its overlapping row, and sends the data issued from internal domain corresponding to the overlapping row of the other processor.
       

\colorbox{yellow}{Figure IV.4: pencil splitting with the additional outer halos }


   The OPA model computes equation terms with the help of mask arrays ( 0 onto land points and 1 onto sea points). It is easily readable and very efficient in the context of the vectorial architecture. But in the case of scalar processor, computations over the land regions becomes more expensive in term of CPU time. It is all the more when we use a complex configuration with a realistic bathymetry like the global ocean where more than 50 \% of points are land points. For this reason, a pre-processing tool allows to search in the mpp domain decomposition strategy if a splitting can be found with a maximum number of only land points processors which could be eliminated (mppini2 program). This optimisation is made with the knowledge of the specific bathymetry in a first time and after, the OPA model, in its initialization part, take account only processors with a sea region. For that, one must indicate in the parameter file the initial cutting along i- and j-axes with jpni and jpnjand the ocean processor number jpnij < jpni x jpnj. Each processor name and neighbour parameters (nbound, nono, noea,...) are modified by an algorithm in the inimpp2.F subroutine.

   The OPA model computes equation terms with the help of mask arrays (0 onto land points and 1 onto sea points). It is easily readable and very efficient in the context of the vectorial architecture. But in the case of scalar processor, computations over the land regions becomes more expensive in term of CPU time. It is all the more so when we use a complex configuration with a realistic bathymetry like the global ocean where more than 50 \% of points are land points. For this reason, a pre-processing tool allows to search in the mpp domain decomposition strategy if a splitting can be found with a maximum number of only land points processors which could be eliminated: the mpp\_optimiz tools (available from the DRAKKAR web site). This optimisation is made with the knowledge of the specific bathymetry. The user chooses optimal parameters \jp{jpni}, \jp{jpnj} and \jp{jpnij} with $jpnij < jpni \times jpnj$, leading to the elimination of $jpni \times jpnj - jpnij$ land processors. When those parameters are specified in module \mdl{par\_oce}, the algorithm in the \rou{inimpp2} routine set each processor name and neighbour parameters (nbound, nono, noea,...) so that the land processors are not taken into account. 

\colorbox{yellow}{Note that the inimpp2 routine is general so that the original inimpp routine should be suppressed from the code.}

When land processors are eliminated, the value corresponding to these locations in the model output files is zero. Note that this is a problem for a mesh output file written by such a model configuration, because model users often divide by the scale factors ($e1t$, $e2t$, etc) and do not expect the grid size to be zero, even on land. It may be best not to eliminate land processors when running the model especially to write the mesh files as outputs (when \np{nmsh} namelist parameter differs from 0).

%>>>>>>>>>>>>>>>>>>>>>>>>>>>>
\begin{figure}[!ht] \label{Fig_mppini2}  \begin{center}
\includegraphics[width=0.90\textwidth]{./Figures/Fig_mppini2.pdf}
\caption {Example of Atlantic domain defined for the CLIPPER projet. Initial grid is composed by 773 x 1236 horizontal points. (a) the domain is splitting onto 9 \time 20 subdomains (jpni=9, jpnj=20). 52 subdomains are land areas. (b) 52 subdomains are eliminated (white rectangles) and the resulting number of processors really used during the computation is jpnij=128.}
\end{center}   \end{figure}
%>>>>>>>>>>>>>>>>>>>>>>>>>>>>


% ================================================================
% Open Boundary Conditions 
% ================================================================
\section{Open Boundary Conditions (\key{obc})}
\label{LBC_obc}

%gm%  to be update from documentation already written by J.M. Molines + Mercator input

% ================================================================
% Flow Relaxation Scheme 
% ================================================================
\section{Flow Relaxation Scheme (???)}
\label{LBC_bdy}

%gm% to be updated by Met Office