[707] | 1 | % ================================================================ |
---|
[4147] | 2 | % Chapter � Miscellaneous Topics |
---|
[707] | 3 | % ================================================================ |
---|
[2282] | 4 | \chapter{Miscellaneous Topics} |
---|
[707] | 5 | \label{MISC} |
---|
| 6 | \minitoc |
---|
| 7 | |
---|
[2282] | 8 | \newpage |
---|
| 9 | $\ $\newline % force a new ligne |
---|
| 10 | |
---|
[707] | 11 | % ================================================================ |
---|
| 12 | % Representation of Unresolved Straits |
---|
| 13 | % ================================================================ |
---|
| 14 | \section{Representation of Unresolved Straits} |
---|
| 15 | \label{MISC_strait} |
---|
| 16 | |
---|
[2282] | 17 | In climate modeling, it often occurs that a crucial connections between water masses |
---|
| 18 | is broken as the grid mesh is too coarse to resolve narrow straits. For example, coarse |
---|
| 19 | grid spacing typically closes off the Mediterranean from the Atlantic at the Strait of |
---|
| 20 | Gibraltar. In this case, it is important for climate models to include the effects of salty |
---|
| 21 | water entering the Atlantic from the Mediterranean. Likewise, it is important for the |
---|
| 22 | Mediterranean to replenish its supply of water from the Atlantic to balance the net |
---|
| 23 | evaporation occurring over the Mediterranean region. This problem occurs even in |
---|
[2349] | 24 | eddy permitting simulations. For example, in ORCA 1/4\deg several straits of the Indonesian |
---|
[2282] | 25 | archipelago (Ombai, Lombok...) are much narrow than even a single ocean grid-point. |
---|
[707] | 26 | |
---|
[2282] | 27 | We describe briefly here the three methods that can be used in \NEMO to handle |
---|
| 28 | such improperly resolved straits. The first two consist of opening the strait by hand |
---|
| 29 | while ensuring that the mass exchanges through the strait are not too large by |
---|
| 30 | either artificially reducing the surface of the strait grid-cells or, locally increasing |
---|
| 31 | the lateral friction. In the third one, the strait is closed but exchanges of mass, |
---|
| 32 | heat and salt across the land are allowed. |
---|
| 33 | Note that such modifications are so specific to a given configuration that no attempt |
---|
| 34 | has been made to set them in a generic way. However, examples of how |
---|
[4147] | 35 | they can be set up is given in the ORCA 2\deg and 0.5\deg configurations. For example, |
---|
| 36 | for details of implementation in ORCA2, search: |
---|
| 37 | \vspace{-10pt} |
---|
| 38 | \begin{alltt} |
---|
| 39 | \tiny |
---|
| 40 | \begin{verbatim} |
---|
| 41 | IF( cp_cfg == "orca" .AND. jp_cfg == 2 ) |
---|
| 42 | \end{verbatim} |
---|
| 43 | \end{alltt} |
---|
[707] | 44 | |
---|
| 45 | % ------------------------------------------------------------------------------------------------------------- |
---|
| 46 | % Hand made geometry changes |
---|
| 47 | % ------------------------------------------------------------------------------------------------------------- |
---|
| 48 | \subsection{Hand made geometry changes} |
---|
| 49 | \label{MISC_strait_hand} |
---|
| 50 | |
---|
[2282] | 51 | $\bullet$ reduced scale factor in the cross-strait direction to a value in better agreement |
---|
| 52 | with the true mean width of the strait. (Fig.~\ref{Fig_MISC_strait_hand}). |
---|
| 53 | This technique is sometime called "partially open face" or "partially closed cells". |
---|
| 54 | The key issue here is only to reduce the faces of $T$-cell ($i.e.$ change the value |
---|
| 55 | of the horizontal scale factors at $u$- or $v$-point) but not the volume of the $T$-cell. |
---|
| 56 | Indeed, reducing the volume of strait $T$-cell can easily produce a numerical |
---|
| 57 | instability at that grid point that would require a reduction of the model time step. |
---|
| 58 | The changes associated with strait management are done in \mdl{domhgr}, |
---|
| 59 | just after the definition or reading of the horizontal scale factors. |
---|
[707] | 60 | |
---|
[2282] | 61 | $\bullet$ increase of the viscous boundary layer thickness by local increase of the |
---|
| 62 | fmask value at the coast (Fig.~\ref{Fig_MISC_strait_hand}). This is done in |
---|
| 63 | \mdl{dommsk} together with the setting of the coastal value of fmask |
---|
| 64 | (see Section \ref{LBC_coast}) |
---|
[994] | 65 | |
---|
[707] | 66 | %>>>>>>>>>>>>>>>>>>>>>>>>>>>> |
---|
[2376] | 67 | \begin{figure}[!tbp] \begin{center} |
---|
[998] | 68 | \includegraphics[width=0.80\textwidth]{./TexFiles/Figures/Fig_Gibraltar.pdf} |
---|
| 69 | \includegraphics[width=0.80\textwidth]{./TexFiles/Figures/Fig_Gibraltar2.pdf} |
---|
[2376] | 70 | \caption{ \label{Fig_MISC_strait_hand} |
---|
| 71 | Example of the Gibraltar strait defined in a $1\deg \times 1\deg$ mesh. |
---|
[994] | 72 | \textit{Top}: using partially open cells. The meridional scale factor at $v$-point |
---|
| 73 | is reduced on both sides of the strait to account for the real width of the strait |
---|
[1224] | 74 | (about 20 km). Note that the scale factors of the strait $T$-point remains unchanged. |
---|
| 75 | \textit{Bottom}: using viscous boundary layers. The four fmask parameters |
---|
[994] | 76 | along the strait coastlines are set to a value larger than 4, $i.e.$ "strong" no-slip |
---|
| 77 | case (see Fig.\ref{Fig_LBC_shlat}) creating a large viscous boundary layer |
---|
| 78 | that allows a reduced transport through the strait.} |
---|
[707] | 79 | \end{center} \end{figure} |
---|
| 80 | %>>>>>>>>>>>>>>>>>>>>>>>>>>>> |
---|
| 81 | |
---|
| 82 | % ------------------------------------------------------------------------------------------------------------- |
---|
| 83 | % Cross Land Advection |
---|
| 84 | % ------------------------------------------------------------------------------------------------------------- |
---|
[2282] | 85 | \subsection{Cross Land Advection (\mdl{tracla})} |
---|
[707] | 86 | \label{MISC_strait_cla} |
---|
[1225] | 87 | %--------------------------------------------namcla-------------------------------------------------------- |
---|
| 88 | \namdisplay{namcla} |
---|
[707] | 89 | %-------------------------------------------------------------------------------------------------------------- |
---|
| 90 | |
---|
| 91 | \colorbox{yellow}{Add a short description of CLA staff here or in lateral boundary condition chapter?} |
---|
[4147] | 92 | Options are defined through the \ngn{namcla} namelist variables. |
---|
[707] | 93 | |
---|
[2282] | 94 | %The problem is resolved here by allowing the mixing of tracers and mass/volume between non-adjacent water columns at nominated regions within the model. Momentum is not mixed. The scheme conserves total tracer content, and total volume (the latter in $z*$- or $s*$-coordinate), and maintains compatibility between the tracer and mass/volume budgets. |
---|
| 95 | |
---|
[707] | 96 | % ================================================================ |
---|
| 97 | % Closed seas |
---|
| 98 | % ================================================================ |
---|
[2282] | 99 | \section{Closed seas (\mdl{closea})} |
---|
[707] | 100 | \label{MISC_closea} |
---|
| 101 | |
---|
[2282] | 102 | \colorbox{yellow}{Add here a short description of the way closed seas are managed} |
---|
[707] | 103 | |
---|
[2282] | 104 | |
---|
[707] | 105 | % ================================================================ |
---|
| 106 | % Sub-Domain Functionality (\textit{nizoom, njzoom}, namelist parameters) |
---|
| 107 | % ================================================================ |
---|
[4147] | 108 | \section{Sub-Domain Functionality (\np{jpizoom}, \np{jpjzoom})} |
---|
[707] | 109 | \label{MISC_zoom} |
---|
| 110 | |
---|
[994] | 111 | The sub-domain functionality, also improperly called the zoom option |
---|
| 112 | (improperly because it is not associated with a change in model resolution) |
---|
| 113 | is a quite simple function that allows a simulation over a sub-domain of an |
---|
| 114 | already defined configuration ($i.e.$ without defining a new mesh, initial |
---|
| 115 | state and forcings). This option can be useful for testing the user settings |
---|
| 116 | of surface boundary conditions, or the initial ocean state of a huge ocean |
---|
| 117 | model configuration while having a small computer memory requirement. |
---|
| 118 | It can also be used to easily test specific physics in a sub-domain (for example, |
---|
[2282] | 119 | see \citep{Madec_al_JPO96} for a test of the coupling used in the global ocean |
---|
[994] | 120 | version of OPA between sea-ice and ocean model over the Arctic or Antarctic |
---|
| 121 | ocean, using a sub-domain). In the standard model, this option does not |
---|
| 122 | include any specific treatment for the ocean boundaries of the sub-domain: |
---|
| 123 | they are considered as artificial vertical walls. Nevertheless, it is quite easy |
---|
| 124 | to add a restoring term toward a climatology in the vicinity of such boundaries |
---|
| 125 | (see \S\ref{TRA_dmp}). |
---|
[707] | 126 | |
---|
| 127 | In order to easily define a sub-domain over which the computation can be |
---|
| 128 | performed, the dimension of all input arrays (ocean mesh, bathymetry, |
---|
[4147] | 129 | forcing, initial state, ...) are defined as \np{jpidta}, \np{jpjdta} and \np{jpkdta} |
---|
| 130 | ( in \ngn{namcfg} namelist), while the computational domain is defined through |
---|
| 131 | \np{jpiglo}, \np{jpjglo} and \jp{jpk} (\ngn{namcfg} namelist). When running the |
---|
| 132 | model over the whole domain, the user sets \np{jpiglo}=\np{jpidta} \np{jpjglo}=\np{jpjdta} |
---|
[994] | 133 | and \jp{jpk}=\jp{jpkdta}. When running the model over a sub-domain, the user |
---|
[4147] | 134 | has to provide the size of the sub-domain, (\np{jpiglo}, \np{jpjglo}, \np{jpkglo}), |
---|
| 135 | and the indices of the south western corner as \np{jpizoom} and \np{jpjzoom} in |
---|
| 136 | the \ngn{namcfg} namelist (Fig.~\ref{Fig_LBC_zoom}). |
---|
[707] | 137 | |
---|
[994] | 138 | Note that a third set of dimensions exist, \jp{jpi}, \jp{jpj} and \jp{jpk} which is |
---|
[4147] | 139 | actually used to perform the computation. It is set by default to \jp{jpi}=\np{jpjglo} |
---|
| 140 | and \jp{jpj}=\np{jpjglo}, except for massively parallel computing where the |
---|
[994] | 141 | computational domain is laid out on local processor memories following a 2D |
---|
| 142 | horizontal splitting. % (see {\S}IV.2-c) ref to the section to be updated |
---|
[707] | 143 | |
---|
| 144 | %>>>>>>>>>>>>>>>>>>>>>>>>>>>> |
---|
[2376] | 145 | \begin{figure}[!ht] \begin{center} |
---|
[998] | 146 | \includegraphics[width=0.90\textwidth]{./TexFiles/Figures/Fig_LBC_zoom.pdf} |
---|
[2376] | 147 | \caption{ \label{Fig_LBC_zoom} |
---|
| 148 | Position of a model domain compared to the data input domain when the zoom functionality is used.} |
---|
[707] | 149 | \end{center} \end{figure} |
---|
| 150 | %>>>>>>>>>>>>>>>>>>>>>>>>>>>> |
---|
| 151 | |
---|
| 152 | |
---|
| 153 | % ================================================================ |
---|
| 154 | % Accelerating the Convergence |
---|
| 155 | % ================================================================ |
---|
| 156 | \section{Accelerating the Convergence (\np{nn\_acc} = 1)} |
---|
| 157 | \label{MISC_acc} |
---|
| 158 | %--------------------------------------------namdom------------------------------------------------------- |
---|
| 159 | \namdisplay{namdom} |
---|
| 160 | %-------------------------------------------------------------------------------------------------------------- |
---|
| 161 | |
---|
[2282] | 162 | Searching an equilibrium state with an global ocean model requires a very long time |
---|
| 163 | integration period (a few thousand years for a global model). Due to the size of |
---|
| 164 | the time step required for numerical stability (less than a few hours), |
---|
| 165 | this usually requires a large elapsed time. In order to overcome this problem, |
---|
| 166 | \citet{Bryan1984} introduces a technique that is intended to accelerate |
---|
[994] | 167 | the spin up to equilibrium. It uses a larger time step in |
---|
[2282] | 168 | the tracer evolution equations than in the momentum evolution |
---|
[707] | 169 | equations. It does not affect the equilibrium solution but modifies the |
---|
| 170 | trajectory to reach it. |
---|
| 171 | |
---|
[4147] | 172 | Options are defined through the \ngn{namdom} namelist variables. |
---|
[994] | 173 | The acceleration of convergence option is used when \np{nn\_acc}=1. In that case, |
---|
[2282] | 174 | $\rdt=rn\_rdt$ is the time step of dynamics while $\widetilde{\rdt}=rdttra$ is the |
---|
| 175 | tracer time-step. the former is set from the \np{rn\_rdt} namelist parameter while the latter |
---|
| 176 | is computed using a hyperbolic tangent profile and the following namelist parameters : |
---|
| 177 | \np{rn\_rdtmin}, \np{rn\_rdtmax} and \np{rn\_rdth}. Those three parameters correspond |
---|
| 178 | to the surface value the deep ocean value and the depth at which the transition occurs, respectively. |
---|
| 179 | The set of prognostic equations to solve becomes: |
---|
[707] | 180 | \begin{equation} \label{Eq_acc} |
---|
| 181 | \begin{split} |
---|
| 182 | \frac{\partial \textbf{U}_h }{\partial t} |
---|
[2282] | 183 | &\equiv \frac{\textbf{U}_h ^{t+1}-\textbf{U}_h^{t-1} }{2\rdt} = \ldots \\ |
---|
| 184 | \frac{\partial T}{\partial t} &\equiv \frac{T^{t+1}-T^{t-1}}{2 \widetilde{\rdt}} = \ldots \\ |
---|
| 185 | \frac{\partial S}{\partial t} &\equiv \frac{S^{t+1} -S^{t-1}}{2 \widetilde{\rdt}} = \ldots \\ |
---|
[707] | 186 | \end{split} |
---|
| 187 | \end{equation} |
---|
| 188 | |
---|
[994] | 189 | \citet{Bryan1984} has examined the consequences of this distorted physics. |
---|
| 190 | Free waves have a slower phase speed, their meridional structure is slightly |
---|
[707] | 191 | modified, and the growth rate of baroclinically unstable waves is reduced |
---|
[994] | 192 | but with a wider range of instability. This technique is efficient for |
---|
| 193 | searching for an equilibrium state in coarse resolution models. However its |
---|
[707] | 194 | application is not suitable for many oceanic problems: it cannot be used for |
---|
| 195 | transient or time evolving problems (in particular, it is very questionable |
---|
[994] | 196 | to use this technique when there is a seasonal cycle in the forcing fields), |
---|
[707] | 197 | and it cannot be used in high-resolution models where baroclinically |
---|
| 198 | unstable processes are important. Moreover, the vertical variation of |
---|
[2282] | 199 | $\widetilde{ \rdt}$ implies that the heat and salt contents are no longer |
---|
[707] | 200 | conserved due to the vertical coupling of the ocean level through both |
---|
[2282] | 201 | advection and diffusion. Therefore \np{rn\_rdtmin} = \np{rn\_rdtmax} should be |
---|
| 202 | a more clever choice. |
---|
[707] | 203 | |
---|
[2541] | 204 | |
---|
[707] | 205 | % ================================================================ |
---|
[2541] | 206 | % Accuracy and Reproducibility |
---|
| 207 | % ================================================================ |
---|
| 208 | \section{Accuracy and Reproducibility (\mdl{lib\_fortran})} |
---|
| 209 | \label{MISC_fortran} |
---|
| 210 | |
---|
| 211 | \subsection{Issues with intrinsinc SIGN function (\key{nosignedzero})} |
---|
| 212 | \label{MISC_sign} |
---|
| 213 | |
---|
| 214 | The SIGN(A, B) is the \textsc {Fortran} intrinsic function delivers the magnitude |
---|
| 215 | of A with the sign of B. For example, SIGN(-3.0,2.0) has the value 3.0. |
---|
| 216 | The problematic case is when the second argument is zero, because, on platforms |
---|
| 217 | that support IEEE arithmetic, zero is actually a signed number. |
---|
| 218 | There is a positive zero and a negative zero. |
---|
| 219 | |
---|
| 220 | In \textsc{Fortran}~90, the processor was required always to deliver a positive result for SIGN(A, B) |
---|
| 221 | if B was zero. Nevertheless, in \textsc{Fortran}~95, the processor is allowed to do the correct thing |
---|
| 222 | and deliver ABS(A) when B is a positive zero and -ABS(A) when B is a negative zero. |
---|
| 223 | This change in the specification becomes apparent only when B is of type real, and is zero, |
---|
| 224 | and the processor is capable of distinguishing between positive and negative zero, |
---|
| 225 | and B is negative real zero. Then SIGN delivers a negative result where, under \textsc{Fortran}~90 |
---|
| 226 | rules, it used to return a positive result. |
---|
| 227 | This change may be especially sensitive for the ice model, so we overwrite the intrinsinc |
---|
| 228 | function with our own function simply performing : \\ |
---|
| 229 | \verb? IF( B >= 0.e0 ) THEN ; SIGN(A,B) = ABS(A) ? \\ |
---|
| 230 | \verb? ELSE ; SIGN(A,B) =-ABS(A) ? \\ |
---|
| 231 | \verb? ENDIF ? \\ |
---|
| 232 | This feature can be found in \mdl{lib\_fortran} module and is effective when \key{nosignedzero} |
---|
| 233 | is defined. We use a CPP key as the overwritting of a intrinsic function can present |
---|
| 234 | performance issues with some computers/compilers. |
---|
| 235 | |
---|
| 236 | |
---|
| 237 | \subsection{MPP reproducibility} |
---|
| 238 | \label{MISC_glosum} |
---|
| 239 | |
---|
| 240 | The numerical reproducibility of simulations on distributed memory parallel computers |
---|
| 241 | is a critical issue. In particular, within NEMO global summation of distributed arrays |
---|
| 242 | is most susceptible to rounding errors, and their propagation and accumulation cause |
---|
| 243 | uncertainty in final simulation reproducibility on different numbers of processors. |
---|
| 244 | To avoid so, based on \citet{He_Ding_JSC01} review of different technics, |
---|
| 245 | we use a so called self-compensated summation method. The idea is to estimate |
---|
| 246 | the roundoff error, store it in a buffer, and then add it back in the next addition. |
---|
| 247 | |
---|
| 248 | Suppose we need to calculate $b = a_1 + a_2 + a_3$. The following algorithm |
---|
| 249 | will allow to split the sum in two ($sum_1 = a_{1} + a_{2}$ and $b = sum_2 = sum_1 + a_3$) |
---|
| 250 | with exactly the same rounding errors as the sum performed all at once. |
---|
| 251 | \begin{align*} |
---|
| 252 | sum_1 \ \ &= a_1 + a_2 \\ |
---|
| 253 | error_1 &= a_2 + ( a_1 - sum_1 ) \\ |
---|
| 254 | sum_2 \ \ &= sum_1 + a_3 + error_1 \\ |
---|
| 255 | error_2 &= a_3 + error_1 + ( sum_1 - sum_2 ) \\ |
---|
| 256 | b \qquad \ &= sum_2 \\ |
---|
| 257 | \end{align*} |
---|
| 258 | This feature can be found in \mdl{lib\_fortran} module and is effective when \key{mpp\_rep}. |
---|
| 259 | In that case, all calls to glob\_sum function (summation over the entire basin excluding |
---|
| 260 | duplicated rows and columns due to cyclic or north fold boundary condition as well as |
---|
| 261 | overlap MPP areas). |
---|
| 262 | Note this implementation may be sensitive to the optimization level. |
---|
| 263 | |
---|
[3294] | 264 | \subsection{MPP scalability} |
---|
| 265 | \label{MISC_mppsca} |
---|
[2541] | 266 | |
---|
[3294] | 267 | The default method of communicating values across the north-fold in distributed memory applications |
---|
| 268 | (\key{mpp\_mpi}) uses a \textsc{MPI\_ALLGATHER} function to exchange values from each processing |
---|
| 269 | region in the northern row with every other processing region in the northern row. This enables a |
---|
| 270 | global width array containing the top 4 rows to be collated on every northern row processor and then |
---|
| 271 | folded with a simple algorithm. Although conceptually simple, this "All to All" communication will |
---|
| 272 | hamper performance scalability for large numbers of northern row processors. From version 3.4 |
---|
| 273 | onwards an alternative method is available which only performs direct "Peer to Peer" communications |
---|
| 274 | between each processor and its immediate "neighbours" across the fold line. This is achieved by |
---|
| 275 | using the default \textsc{MPI\_ALLGATHER} method during initialisation to help identify the "active" |
---|
| 276 | neighbours. Stored lists of these neighbours are then used in all subsequent north-fold exchanges to |
---|
| 277 | restrict exchanges to those between associated regions. The collated global width array for each |
---|
| 278 | region is thus only partially filled but is guaranteed to be set at all the locations actually |
---|
| 279 | required by each individual for the fold operation. This alternative method should give identical |
---|
| 280 | results to the default \textsc{ALLGATHER} method and is recommended for large values of \np{jpni}. |
---|
| 281 | The new method is activated by setting \np{ln\_nnogather} to be true ({\bf nammpp}). The |
---|
| 282 | reproducibility of results using the two methods should be confirmed for each new, non-reference |
---|
| 283 | configuration. |
---|
| 284 | |
---|
[2541] | 285 | % ================================================================ |
---|
[707] | 286 | % Model optimisation, Control Print and Benchmark |
---|
| 287 | % ================================================================ |
---|
[994] | 288 | \section{Model Optimisation, Control Print and Benchmark} |
---|
[707] | 289 | \label{MISC_opt} |
---|
[1225] | 290 | %--------------------------------------------namctl------------------------------------------------------- |
---|
| 291 | \namdisplay{namctl} |
---|
[707] | 292 | %-------------------------------------------------------------------------------------------------------------- |
---|
| 293 | |
---|
[2541] | 294 | \gmcomment{why not make these bullets into subsections?} |
---|
[4147] | 295 | Options are defined through the \ngn{namctl} namelist variables. |
---|
[707] | 296 | |
---|
[2349] | 297 | $\bullet$ Vector optimisation: |
---|
[707] | 298 | |
---|
[2282] | 299 | \key{vectopt\_loop} enables the internal loops to collapse. This is very |
---|
| 300 | a very efficient way to increase the length of vector calculations and thus |
---|
| 301 | to speed up the model on vector computers. |
---|
[707] | 302 | |
---|
[994] | 303 | % Add here also one word on NPROMA technique that has been found useless, since compiler have made significant progress during the last decade. |
---|
[707] | 304 | |
---|
[994] | 305 | % Add also one word on NEC specific optimisation (Novercheck option for example) |
---|
[707] | 306 | |
---|
[994] | 307 | $\bullet$ Control print %: describe here 4 things: |
---|
[707] | 308 | |
---|
[994] | 309 | 1- \np{ln\_ctl} : compute and print the trends averaged over the interior domain |
---|
| 310 | in all TRA, DYN, LDF and ZDF modules. This option is very helpful when |
---|
| 311 | diagnosing the origin of an undesired change in model results. |
---|
[707] | 312 | |
---|
[994] | 313 | 2- also \np{ln\_ctl} but using the nictl and njctl namelist parameters to check |
---|
| 314 | the source of differences between mono and multi processor runs. |
---|
[707] | 315 | |
---|
[994] | 316 | 3- \key{esopa} (to be rename key\_nemo) : which is another option for model |
---|
| 317 | management. When defined, this key forces the activation of all options and |
---|
| 318 | CPP keys. For example, all tracer and momentum advection schemes are called! |
---|
[2282] | 319 | Therefore the model results have no physical meaning. |
---|
[994] | 320 | However, this option forces both the compiler and the model to run through |
---|
| 321 | all the \textsc{Fortran} lines of the model. This allows the user to check for obvious |
---|
| 322 | compilation or execution errors with all CPP options, and errors in namelist options. |
---|
[707] | 323 | |
---|
[2282] | 324 | 4- last digit comparison (\np{nn\_bit\_cmp}). In an MPP simulation, the computation of |
---|
[994] | 325 | a sum over the whole domain is performed as the summation over all processors of |
---|
| 326 | each of their sums over their interior domains. This double sum never gives exactly |
---|
| 327 | the same result as a single sum over the whole domain, due to truncation differences. |
---|
| 328 | The "bit comparison" option has been introduced in order to be able to check that |
---|
| 329 | mono-processor and multi-processor runs give exactly the same results. |
---|
[2376] | 330 | %THIS is to be updated with the mpp_sum_glo introduced in v3.3 |
---|
| 331 | % nn_bit_cmp today only check that the nn_cla = 0 (no cross land advection) |
---|
[707] | 332 | |
---|
[2282] | 333 | $\bullet$ Benchmark (\np{nn\_bench}). This option defines a benchmark run based on |
---|
[2376] | 334 | a GYRE configuration (see \S\ref{CFG_gyre}) in which the resolution remains the same |
---|
| 335 | whatever the domain size. This allows a very large model domain to be used, just by |
---|
| 336 | changing the domain size (\jp{jpiglo}, \jp{jpjglo}) and without adjusting either the time-step |
---|
| 337 | or the physical parameterisations. |
---|
[707] | 338 | |
---|
| 339 | |
---|
| 340 | % ================================================================ |
---|
| 341 | % Elliptic solvers (SOL) |
---|
| 342 | % ================================================================ |
---|
[2282] | 343 | \section{Elliptic solvers (SOL)} |
---|
[707] | 344 | \label{MISC_sol} |
---|
| 345 | %--------------------------------------------namdom------------------------------------------------------- |
---|
| 346 | \namdisplay{namsol} |
---|
| 347 | %-------------------------------------------------------------------------------------------------------------- |
---|
| 348 | |
---|
[2282] | 349 | When the filtered sea surface height option is used, the surface pressure gradient is |
---|
| 350 | computed in \mdl{dynspg\_flt}. The force added in the momentum equation is solved implicitely. |
---|
| 351 | It is thus solution of an elliptic equation \eqref{Eq_PE_flt} for which two solvers are available: |
---|
| 352 | a Successive-Over-Relaxation scheme (SOR) and a preconditioned conjugate gradient |
---|
| 353 | scheme(PCG) \citep{Madec_al_OM88, Madec_PhD90}. The solver is selected trough the |
---|
[4147] | 354 | the value of \np{nn\_solv} \ngn{namsol} namelist variable. |
---|
[2282] | 355 | |
---|
[994] | 356 | The PCG is a very efficient method for solving elliptic equations on vector computers. |
---|
| 357 | It is a fast and rather easy method to use; which are attractive features for a large |
---|
| 358 | number of ocean situations (variable bottom topography, complex coastal geometry, |
---|
[2541] | 359 | variable grid spacing, open or cyclic boundaries, etc ...). It does not require |
---|
[994] | 360 | a search for an optimal parameter as in the SOR method. However, the SOR has |
---|
| 361 | been retained because it is a linear solver, which is a very useful property when |
---|
[2282] | 362 | using the adjoint model of \NEMO. |
---|
[707] | 363 | |
---|
[2282] | 364 | At each time step, the time derivative of the sea surface height at time step $t+1$ |
---|
| 365 | (or equivalently the divergence of the \textit{after} barotropic transport) that appears |
---|
| 366 | in the filtering forced is the solution of the elliptic equation obtained from the horizontal |
---|
| 367 | divergence of the vertical summation of \eqref{Eq_PE_flt}. |
---|
| 368 | Introducing the following coefficients: |
---|
| 369 | \begin{equation} \label{Eq_sol_matrix} |
---|
[707] | 370 | \begin{aligned} |
---|
[2282] | 371 | &c_{i,j}^{NS} &&= {2 \rdt }^2 \; \frac{H_v (i,j) \; e_{1v} (i,j)}{e_{2v}(i,j)} \\ |
---|
| 372 | &c_{i,j}^{EW} &&= {2 \rdt }^2 \; \frac{H_u (i,j) \; e_{2u} (i,j)}{e_{1u}(i,j)} \\ |
---|
| 373 | &b_{i,j} &&= \delta_i \left[ e_{2u}M_u \right] - \delta_j \left[ e_{1v}M_v \right]\ , \\ |
---|
[707] | 374 | \end{aligned} |
---|
| 375 | \end{equation} |
---|
[2541] | 376 | the resulting five-point finite difference equation is given by: |
---|
[2282] | 377 | \begin{equation} \label{Eq_solmat} |
---|
| 378 | \begin{split} |
---|
| 379 | c_{i+1,j}^{NS} D_{i+1,j} + \; c_{i,j+1}^{EW} D_{i,j+1} |
---|
| 380 | + c_{i,j} ^{NS} D_{i-1,j} + \; c_{i,j} ^{EW} D_{i,j-1} & \\ |
---|
| 381 | - \left(c_{i+1,j}^{NS} + c_{i,j+1}^{EW} + c_{i,j}^{NS} + c_{i,j}^{EW} \right) D_{i,j} &= b_{i,j} |
---|
| 382 | \end{split} |
---|
| 383 | \end{equation} |
---|
[996] | 384 | \eqref{Eq_solmat} is a linear symmetric system of equations. All the elements of |
---|
| 385 | the corresponding matrix \textbf{A} vanish except those of five diagonals. With |
---|
[707] | 386 | the natural ordering of the grid points (i.e. from west to east and from |
---|
| 387 | south to north), the structure of \textbf{A} is block-tridiagonal with |
---|
| 388 | tridiagonal or diagonal blocks. \textbf{A} is a positive-definite symmetric |
---|
| 389 | matrix of size $(jpi \cdot jpj)^2$, and \textbf{B}, the right hand side of |
---|
| 390 | \eqref{Eq_solmat}, is a vector. |
---|
| 391 | |
---|
[2282] | 392 | Note that in the linear free surface case, the depth that appears in \eqref{Eq_sol_matrix} |
---|
| 393 | does not vary with time, and thus the matrix can be computed once for all. In non-linear free surface |
---|
| 394 | (\key{vvl} defined) the matrix have to be updated at each time step. |
---|
| 395 | |
---|
[707] | 396 | % ------------------------------------------------------------------------------------------------------------- |
---|
| 397 | % Successive Over Relaxation |
---|
| 398 | % ------------------------------------------------------------------------------------------------------------- |
---|
[2282] | 399 | \subsection{Successive Over Relaxation (\np{nn\_solv}=2, \mdl{solsor})} |
---|
[707] | 400 | \label{MISC_solsor} |
---|
| 401 | |
---|
[2282] | 402 | Let us introduce the four cardinal coefficients: |
---|
| 403 | \begin{align*} |
---|
| 404 | a_{i,j}^S &= c_{i,j }^{NS}/d_{i,j} &\qquad a_{i,j}^W &= c_{i,j}^{EW}/d_{i,j} \\ |
---|
| 405 | a_{i,j}^E &= c_{i,j+1}^{EW}/d_{i,j} &\qquad a_{i,j}^N &= c_{i+1,j}^{NS}/d_{i,j} |
---|
| 406 | \end{align*} |
---|
| 407 | where $d_{i,j} = c_{i,j}^{NS}+ c_{i+1,j}^{NS} + c_{i,j}^{EW} + c_{i,j+1}^{EW}$ |
---|
| 408 | (i.e. the diagonal of the matrix). \eqref{Eq_solmat} can be rewritten as: |
---|
| 409 | \begin{equation} \label{Eq_solmat_p} |
---|
| 410 | \begin{split} |
---|
| 411 | a_{i,j}^{N} D_{i+1,j} +\,a_{i,j}^{E} D_{i,j+1} +\, a_{i,j}^{S} D_{i-1,j} +\,a_{i,j}^{W} D_{i,j-1} - D_{i,j} = \tilde{b}_{i,j} |
---|
| 412 | \end{split} |
---|
| 413 | \end{equation} |
---|
| 414 | with $\tilde b_{i,j} = b_{i,j}/d_{i,j}$. \eqref{Eq_solmat_p} is the equation actually solved |
---|
| 415 | with the SOR method. This method used is an iterative one. Its algorithm can be |
---|
| 416 | summarised as follows (see \citet{Haltiner1980} for a further discussion): |
---|
[707] | 417 | |
---|
[994] | 418 | initialisation (evaluate a first guess from previous time step computations) |
---|
[707] | 419 | \begin{equation} |
---|
[2282] | 420 | D_{i,j}^0 = 2 \, D_{i,j}^t - D_{i,j}^{t-1} |
---|
[707] | 421 | \end{equation} |
---|
[994] | 422 | iteration $n$, from $n=0$ until convergence, do : |
---|
[2282] | 423 | \begin{equation} \label{Eq_sor_algo} |
---|
| 424 | \begin{split} |
---|
| 425 | R_{i,j}^n = &a_{i,j}^{N} D_{i+1,j}^n +\,a_{i,j}^{E} D_{i,j+1} ^n |
---|
| 426 | +\, a_{i,j}^{S} D_{i-1,j} ^{n+1}+\,a_{i,j}^{W} D_{i,j-1} ^{n+1} |
---|
| 427 | - D_{i,j}^n - \tilde{b}_{i,j} \\ |
---|
| 428 | D_{i,j} ^{n+1} = &D_{i,j} ^{n} + \omega \;R_{i,j}^n |
---|
| 429 | \end{split} |
---|
[707] | 430 | \end{equation} |
---|
[994] | 431 | where \textit{$\omega $ }satisfies $1\leq \omega \leq 2$. An optimal value exists for |
---|
| 432 | \textit{$\omega$} which significantly accelerates the convergence, but it has to be |
---|
| 433 | adjusted empirically for each model domain (except for a uniform grid where an |
---|
| 434 | analytical expression for \textit{$\omega$} can be found \citep{Richtmyer1967}). |
---|
[2282] | 435 | The value of $\omega$ is set using \np{rn\_sor}, a \textbf{namelist} parameter. |
---|
[994] | 436 | The convergence test is of the form: |
---|
[707] | 437 | \begin{equation} |
---|
[2282] | 438 | \delta = \frac{\sum\limits_{i,j}{R_{i,j}^n}{R_{i,j}^n}} |
---|
| 439 | {\sum\limits_{i,j}{ \tilde{b}_{i,j}^n}{\tilde{b}_{i,j}^n}} \leq \epsilon |
---|
[707] | 440 | \end{equation} |
---|
[994] | 441 | where $\epsilon$ is the absolute precision that is required. It is recommended |
---|
[2282] | 442 | that a value smaller or equal to $10^{-6}$ is used for $\epsilon$ since larger |
---|
| 443 | values may lead to numerically induced basin scale barotropic oscillations. |
---|
| 444 | The precision is specified by setting \np{rn\_eps} (\textbf{namelist} parameter). |
---|
| 445 | In addition, two other tests are used to halt the iterative algorithm. They involve |
---|
| 446 | the number of iterations and the modulus of the right hand side. If the former |
---|
| 447 | exceeds a specified value, \np{nn\_max} (\textbf{namelist} parameter), |
---|
| 448 | or the latter is greater than $10^{15}$, the whole model computation is stopped |
---|
| 449 | and the last computed time step fields are saved in a abort.nc NetCDF file. |
---|
| 450 | In both cases, this usually indicates that there is something wrong in the model |
---|
| 451 | configuration (an error in the mesh, the initial state, the input forcing, |
---|
[994] | 452 | or the magnitude of the time step or of the mixing coefficients). A typical value of |
---|
[2282] | 453 | $nn\_max$ is a few hundred when $\epsilon = 10^{-6}$, increasing to a few |
---|
[994] | 454 | thousand when $\epsilon = 10^{-12}$. |
---|
| 455 | The vectorization of the SOR algorithm is not straightforward. The scheme |
---|
| 456 | contains two linear recurrences on $i$ and $j$. This inhibits the vectorisation. |
---|
[2282] | 457 | \eqref{Eq_sor_algo} can be been rewritten as: |
---|
| 458 | \begin{equation} |
---|
| 459 | \begin{split} |
---|
[707] | 460 | R_{i,j}^n |
---|
[2282] | 461 | = &a_{i,j}^{N} D_{i+1,j}^n +\,a_{i,j}^{E} D_{i,j+1} ^n |
---|
| 462 | +\,a_{i,j}^{S} D_{i-1,j} ^{n}+\,_{i,j}^{W} D_{i,j-1} ^{n} - D_{i,j}^n - \tilde{b}_{i,j} \\ |
---|
| 463 | R_{i,j}^n = &R_{i,j}^n - \omega \;a_{i,j}^{S}\; R_{i,j-1}^n \\ |
---|
| 464 | R_{i,j}^n = &R_{i,j}^n - \omega \;a_{i,j}^{W}\; R_{i-1,j}^n |
---|
| 465 | \end{split} |
---|
[707] | 466 | \end{equation} |
---|
[2282] | 467 | This technique slightly increases the number of iteration required to reach the convergence, |
---|
| 468 | but this is largely compensated by the gain obtained by the suppression of the recurrences. |
---|
[707] | 469 | |
---|
[2282] | 470 | Another technique have been chosen, the so-called red-black SOR. It consist in solving successively |
---|
| 471 | \eqref{Eq_sor_algo} for odd and even grid points. It also slightly reduced the convergence rate |
---|
| 472 | but allows the vectorisation. In addition, and this is the reason why it has been chosen, it is able to handle the north fold boundary condition used in ORCA configuration ($i.e.$ tri-polar global ocean mesh). |
---|
[707] | 473 | |
---|
[2282] | 474 | The SOR method is very flexible and can be used under a wide range of conditions, |
---|
| 475 | including irregular boundaries, interior boundary points, etc. Proofs of convergence, etc. |
---|
| 476 | may be found in the standard numerical methods texts for partial differential equations. |
---|
[707] | 477 | |
---|
| 478 | % ------------------------------------------------------------------------------------------------------------- |
---|
| 479 | % Preconditioned Conjugate Gradient |
---|
| 480 | % ------------------------------------------------------------------------------------------------------------- |
---|
[2282] | 481 | \subsection{Preconditioned Conjugate Gradient (\np{nn\_solv}=1, \mdl{solpcg}) } |
---|
[707] | 482 | \label{MISC_solpcg} |
---|
| 483 | |
---|
| 484 | \textbf{A} is a definite positive symmetric matrix, thus solving the linear |
---|
[994] | 485 | system \eqref{Eq_solmat} is equivalent to the minimisation of a quadratic |
---|
[707] | 486 | functional: |
---|
| 487 | \begin{equation*} |
---|
| 488 | \textbf{Ax} = \textbf{b} \leftrightarrow \textbf{x} =\text{inf}_{y} \,\phi (\textbf{y}) |
---|
| 489 | \quad , \qquad |
---|
| 490 | \phi (\textbf{y}) = 1/2 \langle \textbf{Ay},\textbf{y}\rangle - \langle \textbf{b},\textbf{y} \rangle |
---|
| 491 | \end{equation*} |
---|
| 492 | where $\langle , \rangle$ is the canonical dot product. The idea of the |
---|
[994] | 493 | conjugate gradient method is to search for the solution in the following |
---|
| 494 | iterative way: assuming that $\textbf{x}^n$ has been obtained, $\textbf{x}^{n+1}$ |
---|
| 495 | is found from $\textbf {x}^{n+1}={\textbf {x}}^n+\alpha^n{\textbf {d}}^n$ which satisfies: |
---|
[707] | 496 | \begin{equation*} |
---|
[994] | 497 | {\textbf{ x}}^{n+1}=\text{inf} _{{\textbf{ y}}={\textbf{ x}}^n+\alpha^n \,{\textbf{ d}}^n} \,\phi ({\textbf{ y}})\;\;\Leftrightarrow \;\;\frac{d\phi }{d\alpha}=0 |
---|
[707] | 498 | \end{equation*} |
---|
[1224] | 499 | and expressing $\phi (\textbf{y})$ as a function of \textit{$\alpha $}, we obtain the |
---|
| 500 | value that minimises the functional: |
---|
[707] | 501 | \begin{equation*} |
---|
| 502 | \alpha ^n = \langle{ \textbf{r}^n , \textbf{r}^n} \rangle / \langle {\textbf{ A d}^n, \textbf{d}^n} \rangle |
---|
| 503 | \end{equation*} |
---|
[994] | 504 | where $\textbf{r}^n = \textbf{b}-\textbf{A x}^n = \textbf{A} (\textbf{x}-\textbf{x}^n)$ |
---|
| 505 | is the error at rank $n$. The descent vector $\textbf{d}^n$ s chosen to be dependent |
---|
| 506 | on the error: $\textbf{d}^n = \textbf{r}^n + \beta^n \,\textbf{d}^{n-1}$. $\beta ^n$ |
---|
| 507 | is searched such that the descent vectors form an orthogonal basis for the dot |
---|
| 508 | product linked to \textbf{A}. Expressing the condition |
---|
[707] | 509 | $\langle \textbf{A d}^n, \textbf{d}^{n-1} \rangle = 0$ the value of $\beta ^n$ is found: |
---|
[1224] | 510 | $\beta ^n = \langle{ \textbf{r}^n , \textbf{r}^n} \rangle / \langle {\textbf{r}^{n-1}, \textbf{r}^{n-1}} \rangle$. |
---|
| 511 | As a result, the errors $ \textbf{r}^n$ form an orthogonal |
---|
[994] | 512 | base for the canonic dot product while the descent vectors $\textbf{d}^n$ form |
---|
[707] | 513 | an orthogonal base for the dot product linked to \textbf{A}. The resulting |
---|
| 514 | algorithm is thus the following one: |
---|
| 515 | |
---|
| 516 | initialisation : |
---|
| 517 | \begin{equation*} |
---|
[2282] | 518 | \begin{split} |
---|
| 519 | \textbf{x}^0 &= D_{i,j}^0 = 2 D_{i,j}^t - D_{i,j}^{t-1} \quad, \text{the initial guess } \\ |
---|
| 520 | \textbf{r}^0 &= \textbf{d}^0 = \textbf{b} - \textbf{A x}^0 \\ |
---|
| 521 | \gamma_0 &= \langle{ \textbf{r}^0 , \textbf{r}^0} \rangle |
---|
| 522 | \end{split} |
---|
[707] | 523 | \end{equation*} |
---|
| 524 | |
---|
| 525 | iteration $n,$ from $n=0$ until convergence, do : |
---|
| 526 | \begin{equation} |
---|
| 527 | \begin{split} |
---|
| 528 | \text{z}^n& = \textbf{A d}^n \\ |
---|
[2282] | 529 | \alpha_n &= \gamma_n / \langle{ \textbf{z}^n , \textbf{d}^n} \rangle \\ |
---|
[707] | 530 | \textbf{x}^{n+1} &= \textbf{x}^n + \alpha_n \,\textbf{d}^n \\ |
---|
| 531 | \textbf{r}^{n+1} &= \textbf{r}^n - \alpha_n \,\textbf{z}^n \\ |
---|
| 532 | \gamma_{n+1} &= \langle{ \textbf{r}^{n+1} , \textbf{r}^{n+1}} \rangle \\ |
---|
| 533 | \beta_{n+1} &= \gamma_{n+1}/\gamma_{n} \\ |
---|
| 534 | \textbf{d}^{n+1} &= \textbf{r}^{n+1} + \beta_{n+1}\; \textbf{d}^{n}\\ |
---|
| 535 | \end{split} |
---|
| 536 | \end{equation} |
---|
| 537 | |
---|
| 538 | |
---|
| 539 | The convergence test is: |
---|
| 540 | \begin{equation} |
---|
| 541 | \delta = \gamma_{n}\; / \langle{ \textbf{b} , \textbf{b}} \rangle \leq \epsilon |
---|
| 542 | \end{equation} |
---|
| 543 | where $\epsilon $ is the absolute precision that is required. As for the SOR algorithm, |
---|
[2282] | 544 | the whole model computation is stopped when the number of iterations, \np{nn\_max}, or |
---|
[994] | 545 | the modulus of the right hand side of the convergence equation exceeds a |
---|
| 546 | specified value (see \S\ref{MISC_solsor} for a further discussion). The required |
---|
| 547 | precision and the maximum number of iterations allowed are specified by setting |
---|
[2282] | 548 | \np{rn\_eps} and \np{nn\_max} (\textbf{namelist} parameters). |
---|
[707] | 549 | |
---|
[994] | 550 | It can be demonstrated that the above algorithm is optimal, provides the exact |
---|
[707] | 551 | solution in a number of iterations equal to the size of the matrix, and that |
---|
[994] | 552 | the convergence rate is faster as the matrix is closer to the identity matrix, |
---|
[2282] | 553 | $i.e.$ its eigenvalues are closer to 1. Therefore, it is more efficient to solve |
---|
| 554 | a better conditioned system which has the same solution. For that purpose, |
---|
| 555 | we introduce a preconditioning matrix \textbf{Q} which is an approximation |
---|
| 556 | of \textbf{A} but much easier to invert than \textbf{A}, and solve the system: |
---|
[994] | 557 | \begin{equation} \label{Eq_pmat} |
---|
[707] | 558 | \textbf{Q}^{-1} \textbf{A x} = \textbf{Q}^{-1} \textbf{b} |
---|
| 559 | \end{equation} |
---|
| 560 | |
---|
[994] | 561 | The same algorithm can be used to solve \eqref{Eq_pmat} if instead of the |
---|
| 562 | canonical dot product the following one is used: |
---|
[1224] | 563 | ${\langle{ \textbf{a} , \textbf{b}} \rangle}_Q = \langle{ \textbf{a} , \textbf{Q b}} \rangle$, and |
---|
| 564 | if $\textbf{\~{b}} = \textbf{Q}^{-1}\;\textbf{b}$ and $\textbf{\~{A}} = \textbf{Q}^{-1}\;\textbf{A}$ |
---|
[2282] | 565 | are substituted to \textbf{b} and \textbf{A} \citep{Madec_al_OM88}. |
---|
[1224] | 566 | In \NEMO, \textbf{Q} is chosen as the diagonal of \textbf{ A}, i.e. the simplest form for |
---|
| 567 | \textbf{Q} so that it can be easily inverted. In this case, the discrete formulation of |
---|
[994] | 568 | \eqref{Eq_pmat} is in fact given by \eqref{Eq_solmat_p} and thus the matrix and |
---|
| 569 | right hand side are computed independently from the solver used. |
---|
[707] | 570 | |
---|
| 571 | % ================================================================ |
---|
| 572 | |
---|
| 573 | |
---|
[2282] | 574 | |
---|
[707] | 575 | |
---|
| 576 | |
---|
[2364] | 577 | |
---|
| 578 | |
---|
| 579 | |
---|
| 580 | |
---|
| 581 | |
---|
| 582 | |
---|
| 583 | |
---|