Ignore:
Timestamp:
06/26/18 00:28:38 (3 years ago)
Author:
yushan
Message:

report updated

File:
1 edited

Legend:

Unmodified
Added
Removed
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.tex

    r1548 r1551  
    99% Title Page 
    1010 
    11 \title{Developping XIOS with multithread : to accelerate the IO of climate models} 
     11\title{Developing XIOS with multi-thread : to accelerate the I/O of climate models} 
    1212 
    1313\author{} 
     
    2020 
    2121The simulation models of climate systems, running on a large number of computing resources can produce an important volume of data. At this  
    22 scale, the IO and the post-treatement of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux  
    23 generated by the simulations, we use XIOS developped by the Institut Pierre Simon Laplace and Maison de la simulation. 
     22scale, the I/O and the post-treatment of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux  
     23generated by the simulations, we use XIOS developed by the Institut Pierre Simon Laplace and Maison de la simulation. 
    2424 
    25 XIOS, a libarary dedicated to intense calculates, allows us to easily and efficiently manage the parallel IO on the storage systems. XIOS  
     25XIOS, a library dedicated to intense calculates, allows us to easily and efficiently manage the parallel I/O on the storage systems. XIOS  
    2626uses the client/server scheme in which computing resources (server) are reserved exclusively for IO in order to minimize their impact on  
    27 the performance of the climate models (client). The clients and servers are executed in parallel and communicate asynchronuously. In this  
    28 way, the IO peaks can be smoothed out as data fluxes are send to server constantly throughout the simulation and the time spent on data  
     27the performance of the climate models (client). The clients and servers are executed in parallel and communicate asynchronously. In this  
     28way, the I/O peaks can be smoothed out as data fluxes are send to server constantly throughout the simulation and the time spent on data  
    2929writing on the server side can be overlapped completely by calculates on the client side.  
    3030 
     
    3232\includegraphics[scale=0.4]{Charge1.png} 
    3333\includegraphics[scale=0.4]{Charge2.png} 
    34 \caption{On the left, each peak of computing power corresponds to the vallay of memory bandwidth which shows that the computing resources  
    35 are alternating between calculates and IO. ON the right, both curves are smooth which means that the computing resources have a stable  
    36 charge of work, either calculates or IO.} 
     34\caption{On the left, each peak of computing power corresponds to the valley of memory bandwidth which shows that the computing resources  
     35are alternating between calculates and I/O. ON the right, both curves are smooth which means that the computing resources have a stable  
     36charge of work, either calculates or I/O.} 
    3737\end{figure} 
    3838 
    3939 
    4040XIOS works well with many climate simulation codes. For example, LMDZ\footnote{LMDZ is a general circulation model (or global climate model)  
    41 developped since the 70s at the "Laboratoire de Météorologie Dynamique", which includes various variants for the Earth and other planets  
     41developed since the 70s at the "Laboratoire de Météorologie Dynamique", which includes various variants for the Earth and other planets  
    4242(Mars, Titan, Venus, Exoplanets). The 'Z' in LMDZ stands for "zoom" (and the 'LMD' is for  'Laboratoire de Météorologie Dynamique").  
    43 \url{http://lmdz.lmd.jussieu.fr}}, NENO\footnote{Nucleus for European Modelling of the Ocean alias NEMO is a  
     43\url{http://lmdz.lmd.jussieu.fr}}, NENO\footnote{Nucleus for European Modeling of the Ocean alias NEMO is a  
    4444state-of-the-art modelling framework of ocean related engines. \url{https://www.nemo-ocean.eu}}, ORCHIDEE\footnote{the land surface  
    4545model of the IPSL (Institut Pierre Simon Laplace) Earth System Model. \url{https://orchidee.ipsl.fr}}, and DYNAMICO\footnote{The DYNAMICO  
    4646project develops a new dynamical core for LMD-Z, the atmospheric general circulation model (GCM) part of IPSL-CM Earth System Model.  
    47 \url{http://www.lmd.polytechnique.fr/~dubos/DYNAMICO/}} all use XIOS as the output backend. M\'et\'eoFrance and MetOffice also choose XIOS  
    48 to manege the IO for their models. 
     47\url{http://www.lmd.polytechnique.fr/~dubos/DYNAMICO/}} all use XIOS as the output back end. M\'et\'eoFrance and MetOffice also choose XIOS  
     48to manege the I/O for their models. 
    4949 
    5050 
    51 \section{Developpement of thread-friendly XIOS} 
     51\section{Development of thread-friendly XIOS} 
     52 
     53Although XIOS copes well with many models, there is one potential optimization in XIOS which needs to be investigated: making XIOS thread-friendly. 
     54 
     55This topic comes along with the configuration of the climate models. Take LMDZ as example, it is designed with the 2-level parallelization scheme. To be more specific, LMDZ uses the domain decomposition method in which each sub-domain is associated with one MPI process. Inside of the sub-domain, the model also uses OpenMP derivatives to accelerate the computation. We can imagine that the sub-domain be divided into sub-sub-domain and is managed by threads.  
     56 
     57\begin{figure}[h] 
     58\centering 
     59\includegraphics[scale=0.5]{domain.pdf} 
     60\caption{Illustration of the domain decomposition used in LMDZ.} 
     61\end{figure} 
     62 
     63As we know, each sub-domain, or in another word, each MPI process is a XIOS client. The data exchange between client and XIOS servers is handled by MPI communications. In order to write an output field, all threads must gather the data to the master thread who acts as MPI process in order to call MPI routines. There are two disadvantages about this method : first, we have to spend time on gathering information to the master thread which not only increases the memory use, but also implies an OpenMP barrier; second, while the master thread calls MPI routine, other threads are in the idle state thus a waster of computing resources. What we want obtain with the thread-friendly XIOS is that all threads can act like MPI processes. They can call directly the MPI routine thus no waste in memory nor in computing resources as shown in Figure \ref{fig:omp}. 
     64 
     65\begin{figure}[h!] 
     66\centering 
     67\includegraphics[scale=0.6]{omp.pdf} 
     68\caption{} 
     69\label{fig:omp} 
     70\end{figure} 
     71 
     72There are two ways to make XIOS thread-friendly. First of all, change the structure of XIOS which demands a lot of modification is the XIOS library. Knowing that XIOS is about 100 000 lines of code, this method will be very time consuming. What's more, the modification will be local to XIOS. If we want to optimize an other code to be thread-friendly, we have to redo the modifications. The second choice is to add an extra interface to MPI in order to manage the threads. When a thread want to call an MPI routine inside XIOS, it will first pass the interface, in which the communication information will be analyzed before the MPI routine is invoked. With this method, we only need to modify a very small part of XIOS in order to make it work. What is more interesting is that the interface we created can be adjusted to suit other MPI based libraries. 
    5273 
    5374 
    54 XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations  
    55 and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. 
    56 However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP  
    57 directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to  
    58 maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI  
    59 process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the  
    60 updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines.  
    61 This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a  
    62 new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we  
    63 introduce the MPI endpoints.   
     75In this project, we choose to implement the interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a concept proposed in the last MPI Forum and several papers has already discussed the importance of such idea and have introduced the framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. Threads of an MPI process is associated with a unique rank (global endpoint rank) and an endpoint communicator. They also have a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num| rank.  
     76 
     77\begin{figure}[h!] 
     78\begin{center} 
     79\includegraphics[scale=0.4]{scheme.png}  
     80\end{center} 
     81\caption{} 
     82\label{fig:scheme} 
     83\end{figure} 
     84 
     85%XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we introduce the MPI endpoints.   
    6486 
    6587 
    66 The MPI endpoints (EP) is a layer on top of an existing MPI Implementation. All MPI function, or in our work the functions used in XIOS,  
    67 will be reimplemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be  
     88The MPI\_endpoints interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions used in XIOS.  
     89 
     90will be re-implemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be  
    6891associated with a unique rank and with an endpoint communicator. This rank (EP rank) will replace the role of the classic MPI rank and will  
    6992be used in MPI communications. In order to successfully execute an MPI communication, for example \verb|MPI_Send|, we know already which  
     
    7396 
    7497In XIOS, we used the ``probe'' technique to search for arrived messages and then performing the receive action. The principle is  
    75 that sender processes execute the send operations as usual. However, to minimise the time spent on waiting incoming messages, the receiver  
     98that sender processes execute the send operations as usual. However, to minimize the time spent on waiting incoming messages, the receiver  
    7699processe performs in the first place the \verb|MPI_Probe| function to check if a message destinated to it has been published. If yes, the  
    77100process execute in the second place the \verb|MPI_Recv| to receive the message. In this situation, if we introduce the threads, problems  
     
    135158decrease in time of 25\%. Even the 25\% may seems to be small, it is still a gain in performance with existing computing resources. 
    136159 
     160\section{Performance of EP\_XIOS} 
     161 
     162workfloz\_cmip6  
     163light output 
     16424*8+2 
     16530s - 52s 
     16632 days 
     167histmth with daily output 
     168 
    137169\section{Perspectives of EP\_XIOS} 
    138170 
     171 
     172\bibliographystyle{plain} 
     173\bibliography{reference} 
     174 
    139175\end{document}           
Note: See TracChangeset for help on using the changeset viewer.