Changeset 1552


Ignore:
Timestamp:
06/27/18 15:18:29 (3 years ago)
Author:
yushan
Message:

report updated. Todo: add performance results

Location:
XIOS/dev/branch_openmp/Note
Files:
16 edited

Legend:

Unmodified
Added
Removed
  • XIOS/dev/branch_openmp/Note/EP note.aux

    r1548 r1552  
    2121\@writefile{toc}{\contentsline {section}{\numberline {6}Inter-communicator}{8}} 
    2222\@writefile{toc}{\contentsline {subsection}{\numberline {6.1}The splitting of intra-communicator}{8}} 
    23 \@writefile{toc}{\contentsline {subsection}{\numberline {6.2}The creation of inter-communicator}{10}} 
     23\@writefile{toc}{\contentsline {subsection}{\numberline {6.2}The creation of inter-communicator}{9}} 
    2424\citation{MPI} 
    2525\@writefile{toc}{\contentsline {subsection}{\numberline {6.3}The merge of inter-communicators}{12}} 
  • XIOS/dev/branch_openmp/Note/EP note.log

    r1548 r1552  
    1 This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24)  22 JUN 2018 16:19 
     1This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24)  26 JUN 2018 15:00 
    22entering extended mode 
    33 restricted \write18 enabled. 
     
    638638 
    639639 
    640 Overfull \hbox (25.85847pt too wide) in paragraph at lines 239--240 
     640Overfull \hbox (44.74739pt too wide) in paragraph at lines 239--240 
    641641[]\OT1/cmr/m/n/10 We call []\OT1/cmtt/m/n/10 MPI_Bcast(buffer, count, datatype, 
    642  mpi_root = 1, mpi_comm) \OT1/cmr/m/n/10 .  
    643  [] 
    644  
    645 <bcast.png, id=36, 743.77875pt x 552.0625pt> 
     642 mpi_root = 1, mpi_comm)  \OT1/cmr/m/n/10 with 
     643 [] 
     644 
     645<bcast.png, id=36, 512.91624pt x 552.0625pt> 
    646646File: bcast.png Graphic file (type png) 
    647647 <use bcast.png> 
    648648Package pdftex.def Info: bcast.png used on input line 244. 
    649 (pdftex.def)             Requested size: 223.13535pt x 165.62003pt. 
     649(pdftex.def)             Requested size: 153.87605pt x 165.62003pt. 
    650650 
    651651Overfull \hbox (16.20544pt too wide) in paragraph at lines 249--251 
     
    687687Package pdftex.def Info: intercomm.png used on input line 344. 
    688688(pdftex.def)             Requested size: 311.06451pt x 137.9163pt. 
    689  [10 <./intercomm.png (PNG copy)>] 
    690 <intercomm_step.png, id=56, 1309.89375pt x 683.55376pt> 
     689 
     690<intercomm_step.png, id=53, 1309.89375pt x 683.55376pt> 
    691691File: intercomm_step.png Graphic file (type png) 
    692692 
     
    694694Package pdftex.def Info: intercomm_step.png used on input line 369. 
    695695(pdftex.def)             Requested size: 327.47264pt x 170.88802pt. 
    696  
     696 [10 <./intercomm.png (PNG copy)>] 
    697697Overfull \hbox (6.07089pt too wide) in paragraph at lines 372--375 
    698698\OT1/cmr/m/n/10 should add an \OT1/cmr/m/it/10 pri-or-ity check \OT1/cmr/m/n/10 
     
    827827xlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texlive/ 
    828828texmf-dist/fonts/type1/urw/times/utmr8a.pfb> 
    829 Output written on "EP note.pdf" (17 pages, 1091602 bytes). 
     829Output written on "EP note.pdf" (17 pages, 1089153 bytes). 
    830830PDF statistics: 
    831831 133 PDF objects out of 1000 (max. 8388607) 
  • XIOS/dev/branch_openmp/Note/EP note.tex

    r1548 r1552  
    236236 
    237237\begin{itemize} 
    238  \item[1.] EP process with rank 4 send the buffer to EP process rank 3 which is a master thread.  
    239  \item[2.] We call \verb|MPI_Bcast(buffer, count, datatype, mpi_root = 1, mpi_comm) |. 
    240  \item[3.] All master threads send the buffer to its slaves.  
     238 %\item[1.] EP process with rank 4 send the buffer to EP process rank 3 which is a master thread.  
     239 \item[1.] We call \verb|MPI_Bcast(buffer, count, datatype, mpi_root = 1, mpi_comm) | with EP processes rank 0, 4, 6, and 9. 
     240 \item[2.] EP processes rank 0, 4, 6, and 9 send the buffer to its slaves.  
    241241\end{itemize} 
    242242 
  • XIOS/dev/branch_openmp/Note/bcast.svg

    r1548 r1552  
    44    version="1.1" baseProfile="full"> 
    55    <defs> 
    6         <marker overflow="visible" orient="auto" id="arrow2-3386"> 
     6        <marker overflow="visible" orient="auto" id="arrow2-619"> 
    77            <path 
    88                d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " 
    99                latexdraw:arrSizeNum="3.0" fill="black"/> 
    1010        </marker> 
    11         <marker overflow="visible" orient="auto" id="arrow2-3391"> 
    12             <path 
    13                 d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " 
    14                 latexdraw:arrSizeNum="3.0" fill="black"/> 
    15         </marker> 
    16         <marker overflow="visible" orient="auto" id="arrow2-3442"> 
     11        <marker overflow="visible" orient="auto" id="arrow2-657"> 
    1712            <path 
    1813                d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " 
     
    2116    </defs> 
    2217    <g> 
    23         <g latexdraw:type="drawing" id="id3206"> 
    24             <g latexdraw:type="drawing" id="id3207"> 
    25                 <g latexdraw:type="drawing" id="id3208"> 
    26                     <g latexdraw:type="rectangle" id="id3209"> 
    27                         <rect x="351.0" y="531.0" width="38.0" 
    28                             height="118.0" latexdraw:borderPos="outer" 
    29                             stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 
    30                     </g> 
    31                     <g latexdraw:type="rectangle" id="id3210"> 
    32                         <rect x="351.0" y="391.0" width="38.0" 
    33                             height="118.0" latexdraw:borderPos="outer" 
    34                             stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 
    35                     </g> 
    36                     <g latexdraw:type="rectangle" id="id3211"> 
    37                         <rect x="351.0" y="251.0" width="38.0" 
    38                             height="118.0" latexdraw:borderPos="outer" 
    39                             stroke-width="2.0" stroke="black" fill="#f1daf9"/> 
    40                     </g> 
    41                     <g latexdraw:type="rectangle" id="id3212"> 
    42                         <rect x="351.0" y="111.0" width="38.0" 
    43                             height="118.0" latexdraw:borderPos="outer" 
    44                             stroke-width="2.0" stroke="black" fill="#fafac9"/> 
    45                     </g> 
    46                 </g> 
    47                 <g latexdraw:type="drawing" id="id3214"> 
    48                     <g latexdraw:type="square" id="id3215"> 
    49                         <rect x="361.0" y="121.0" width="18.0" 
    50                             height="18.0" latexdraw:borderPos="outer" 
    51                             stroke-width="2.0" stroke="black" fill="white"/> 
    52                     </g> 
    53                     <g latexdraw:type="square" id="id3216"> 
    54                         <rect x="361.0" y="161.0" width="18.0" 
    55                             height="18.0" latexdraw:borderPos="outer" 
    56                             stroke-width="2.0" stroke="black" fill="white"/> 
    57                     </g> 
    58                     <g latexdraw:type="square" id="id3217"> 
    59                         <rect x="361.0" y="201.0" width="18.0" 
    60                             height="18.0" latexdraw:borderPos="outer" 
    61                             stroke-width="2.0" stroke="black" fill="white"/> 
    62                     </g> 
    63                 </g> 
    64                 <g latexdraw:type="square" id="id3219"> 
    65                     <rect x="361.0" y="261.0" width="18.0" height="18.0" 
    66                         latexdraw:borderPos="outer" stroke-width="2.0" 
    67                         stroke="black" fill="white"/> 
    68                 </g> 
    69                 <g latexdraw:type="square" id="id3220"> 
    70                     <rect x="361.0" y="301.0" width="18.0" height="18.0" 
    71                         latexdraw:borderPos="outer" stroke-width="2.0" 
    72                         stroke="black" fill="red"/> 
    73                 </g> 
    74                 <g latexdraw:type="square" id="id3221"> 
    75                     <rect x="361.0" y="341.0" width="18.0" height="18.0" 
    76                         latexdraw:borderPos="outer" stroke-width="2.0" 
    77                         stroke="black" fill="white"/> 
    78                 </g> 
    79                 <g latexdraw:type="drawing" id="id3222"> 
    80                     <g latexdraw:type="square" id="id3223"> 
    81                         <rect x="361.0" y="401.0" width="18.0" 
    82                             height="18.0" latexdraw:borderPos="outer" 
    83                             stroke-width="2.0" stroke="black" fill="white"/> 
    84                     </g> 
    85                     <g latexdraw:type="square" id="id3224"> 
    86                         <rect x="361.0" y="441.0" width="18.0" 
    87                             height="18.0" latexdraw:borderPos="outer" 
    88                             stroke-width="2.0" stroke="black" fill="white"/> 
    89                     </g> 
    90                     <g latexdraw:type="square" id="id3225"> 
    91                         <rect x="361.0" y="481.0" width="18.0" 
    92                             height="18.0" latexdraw:borderPos="outer" 
    93                             stroke-width="2.0" stroke="black" fill="white"/> 
    94                     </g> 
    95                 </g> 
    96                 <g latexdraw:type="drawing" id="id3227"> 
    97                     <g latexdraw:type="square" id="id3228"> 
    98                         <rect x="361.0" y="541.0" width="18.0" 
    99                             height="18.0" latexdraw:borderPos="outer" 
    100                             stroke-width="2.0" stroke="black" fill="white"/> 
    101                     </g> 
    102                     <g latexdraw:type="square" id="id3229"> 
    103                         <rect x="361.0" y="581.0" width="18.0" 
    104                             height="18.0" latexdraw:borderPos="outer" 
    105                             stroke-width="2.0" stroke="black" fill="white"/> 
    106                     </g> 
    107                     <g latexdraw:type="square" id="id3230"> 
    108                         <rect x="361.0" y="621.0" width="18.0" 
    109                             height="18.0" latexdraw:borderPos="outer" 
    110                             stroke-width="2.0" stroke="black" fill="white"/> 
    111                     </g> 
    112                 </g> 
    113                 <g latexdraw:type="text" id="id3232" 
    114                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    115                     <text x="320.0" y="140.0"><![CDATA[0]]></text> 
    116                 </g> 
    117                 <g latexdraw:type="text" id="id3235" 
    118                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    119                     <text x="320.0" y="180.0"><![CDATA[1]]></text> 
    120                 </g> 
    121                 <g latexdraw:type="text" id="id3238" 
    122                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    123                     <text x="320.0" y="220.0"><![CDATA[2]]></text> 
    124                 </g> 
    125                 <g latexdraw:type="text" id="id3241" 
    126                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    127                     <text x="320.0" y="280.0"><![CDATA[3]]></text> 
    128                 </g> 
    129                 <g latexdraw:type="text" id="id3244" 
    130                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    131                     <text x="320.0" y="320.0"><![CDATA[4]]></text> 
    132                 </g> 
    133                 <g latexdraw:type="text" id="id3247" 
    134                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    135                     <text x="320.0" y="360.0"><![CDATA[5]]></text> 
    136                 </g> 
    137                 <g latexdraw:type="text" id="id3250" 
    138                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    139                     <text x="320.0" y="420.0"><![CDATA[6]]></text> 
    140                 </g> 
    141                 <g latexdraw:type="text" id="id3253" 
    142                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    143                     <text x="320.0" y="460.0"><![CDATA[7]]></text> 
    144                 </g> 
    145                 <g latexdraw:type="text" id="id3256" 
    146                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    147                     <text x="320.0" y="500.0"><![CDATA[8]]></text> 
    148                 </g> 
    149                 <g latexdraw:type="text" id="id3259" 
    150                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    151                     <text x="320.0" y="560.0"><![CDATA[9]]></text> 
    152                 </g> 
    153                 <g latexdraw:type="text" id="id3262" 
    154                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    155                     <text x="320.0" y="600.0"><![CDATA[10]]></text> 
    156                 </g> 
    157                 <g latexdraw:type="text" id="id3265" 
    158                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 
    159                     <text x="320.0" y="640.0"><![CDATA[11]]></text> 
    160                 </g> 
    161             </g> 
    162             <g latexdraw:type="drawing" id="id3269"> 
    163                 <g latexdraw:type="drawing" id="id3270"> 
    164                     <g latexdraw:type="rectangle" id="id3271"> 
    165                         <rect x="571.0" y="531.0" width="38.0" 
    166                             height="118.0" latexdraw:borderPos="outer" 
    167                             stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 
    168                     </g> 
    169                     <g latexdraw:type="rectangle" id="id3272"> 
    170                         <rect x="571.0" y="391.0" width="38.0" 
    171                             height="118.0" latexdraw:borderPos="outer" 
    172                             stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 
    173                     </g> 
    174                     <g latexdraw:type="rectangle" id="id3273"> 
    175                         <rect x="571.0" y="251.0" width="38.0" 
    176                             height="118.0" latexdraw:borderPos="outer" 
    177                             stroke-width="2.0" stroke="black" fill="#f1daf9"/> 
    178                     </g> 
    179                     <g latexdraw:type="rectangle" id="id3274"> 
    180                         <rect x="571.0" y="111.0" width="38.0" 
    181                             height="118.0" latexdraw:borderPos="outer" 
    182                             stroke-width="2.0" stroke="black" fill="#fafac9"/> 
    183                     </g> 
    184                 </g> 
    185                 <g latexdraw:type="drawing" id="id3276"> 
    186                     <g latexdraw:type="square" id="id3277"> 
    187                         <rect x="581.0" y="121.0" width="18.0" 
    188                             height="18.0" latexdraw:borderPos="outer" 
    189                             stroke-width="2.0" stroke="black" fill="white"/> 
    190                     </g> 
    191                     <g latexdraw:type="square" id="id3278"> 
    192                         <rect x="581.0" y="161.0" width="18.0" 
    193                             height="18.0" latexdraw:borderPos="outer" 
    194                             stroke-width="2.0" stroke="black" fill="white"/> 
    195                     </g> 
    196                     <g latexdraw:type="square" id="id3279"> 
    197                         <rect x="581.0" y="201.0" width="18.0" 
    198                             height="18.0" latexdraw:borderPos="outer" 
    199                             stroke-width="2.0" stroke="black" fill="white"/> 
    200                     </g> 
    201                 </g> 
    202                 <g latexdraw:type="square" id="id3281"> 
    203                     <rect x="581.0" y="261.0" width="18.0" height="18.0" 
    204                         latexdraw:borderPos="outer" stroke-width="2.0" 
    205                         stroke="black" fill="red"/> 
    206                 </g> 
    207                 <g latexdraw:type="square" id="id3282"> 
    208                     <rect x="581.0" y="301.0" width="18.0" height="18.0" 
    209                         latexdraw:borderPos="outer" stroke-width="2.0" 
    210                         stroke="black" fill="red"/> 
    211                 </g> 
    212                 <g latexdraw:type="square" id="id3283"> 
    213                     <rect x="581.0" y="341.0" width="18.0" height="18.0" 
    214                         latexdraw:borderPos="outer" stroke-width="2.0" 
    215                         stroke="black" fill="white"/> 
    216                 </g> 
    217                 <g latexdraw:type="drawing" id="id3284"> 
    218                     <g latexdraw:type="square" id="id3285"> 
    219                         <rect x="581.0" y="401.0" width="18.0" 
    220                             height="18.0" latexdraw:borderPos="outer" 
    221                             stroke-width="2.0" stroke="black" fill="white"/> 
    222                     </g> 
    223                     <g latexdraw:type="square" id="id3286"> 
    224                         <rect x="581.0" y="441.0" width="18.0" 
    225                             height="18.0" latexdraw:borderPos="outer" 
    226                             stroke-width="2.0" stroke="black" fill="white"/> 
    227                     </g> 
    228                     <g latexdraw:type="square" id="id3287"> 
    229                         <rect x="581.0" y="481.0" width="18.0" 
    230                             height="18.0" latexdraw:borderPos="outer" 
    231                             stroke-width="2.0" stroke="black" fill="white"/> 
    232                     </g> 
    233                 </g> 
    234                 <g latexdraw:type="drawing" id="id3289"> 
    235                     <g latexdraw:type="square" id="id3290"> 
    236                         <rect x="581.0" y="541.0" width="18.0" 
    237                             height="18.0" latexdraw:borderPos="outer" 
    238                             stroke-width="2.0" stroke="black" fill="white"/> 
    239                     </g> 
    240                     <g latexdraw:type="square" id="id3291"> 
    241                         <rect x="581.0" y="581.0" width="18.0" 
    242                             height="18.0" latexdraw:borderPos="outer" 
    243                             stroke-width="2.0" stroke="black" fill="white"/> 
    244                     </g> 
    245                     <g latexdraw:type="square" id="id3292"> 
    246                         <rect x="581.0" y="621.0" width="18.0" 
    247                             height="18.0" latexdraw:borderPos="outer" 
    248                             stroke-width="2.0" stroke="black" fill="white"/> 
    249                     </g> 
    250                 </g> 
    251                 <g latexdraw:type="text" id="id3294" 
    252                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    253                     <text x="540.0" y="140.0"><![CDATA[0]]></text> 
    254                 </g> 
    255                 <g latexdraw:type="text" id="id3297" 
    256                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    257                     <text x="540.0" y="180.0"><![CDATA[1]]></text> 
    258                 </g> 
    259                 <g latexdraw:type="text" id="id3300" 
    260                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    261                     <text x="540.0" y="220.0"><![CDATA[2]]></text> 
    262                 </g> 
    263                 <g latexdraw:type="text" id="id3303" 
    264                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    265                     <text x="540.0" y="280.0"><![CDATA[3]]></text> 
    266                 </g> 
    267                 <g latexdraw:type="text" id="id3306" 
    268                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    269                     <text x="540.0" y="320.0"><![CDATA[4]]></text> 
    270                 </g> 
    271                 <g latexdraw:type="text" id="id3309" 
    272                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    273                     <text x="540.0" y="360.0"><![CDATA[5]]></text> 
    274                 </g> 
    275                 <g latexdraw:type="text" id="id3312" 
    276                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    277                     <text x="540.0" y="420.0"><![CDATA[6]]></text> 
    278                 </g> 
    279                 <g latexdraw:type="text" id="id3315" 
    280                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    281                     <text x="540.0" y="460.0"><![CDATA[7]]></text> 
    282                 </g> 
    283                 <g latexdraw:type="text" id="id3318" 
    284                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    285                     <text x="540.0" y="500.0"><![CDATA[8]]></text> 
    286                 </g> 
    287                 <g latexdraw:type="text" id="id3321" 
    288                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    289                     <text x="540.0" y="560.0"><![CDATA[9]]></text> 
    290                 </g> 
    291                 <g latexdraw:type="text" id="id3324" 
    292                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    293                     <text x="540.0" y="600.0"><![CDATA[10]]></text> 
    294                 </g> 
    295                 <g latexdraw:type="text" id="id3327" 
    296                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 
    297                     <text x="540.0" y="640.0"><![CDATA[11]]></text> 
    298                 </g> 
    299             </g> 
    300             <g latexdraw:type="drawing" id="id3331"> 
    301                 <g latexdraw:type="drawing" id="id3332"> 
    302                     <g latexdraw:type="rectangle" id="id3333"> 
    303                         <rect x="801.0" y="531.0" width="38.0" 
    304                             height="118.0" latexdraw:borderPos="outer" 
    305                             stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 
    306                     </g> 
    307                     <g latexdraw:type="rectangle" id="id3334"> 
    308                         <rect x="801.0" y="391.0" width="38.0" 
    309                             height="118.0" latexdraw:borderPos="outer" 
    310                             stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 
    311                     </g> 
    312                     <g latexdraw:type="rectangle" id="id3335"> 
    313                         <rect x="801.0" y="251.0" width="38.0" 
    314                             height="118.0" latexdraw:borderPos="outer" 
    315                             stroke-width="2.0" stroke="black" fill="#f1daf9"/> 
    316                     </g> 
    317                     <g latexdraw:type="rectangle" id="id3336"> 
    318                         <rect x="801.0" y="111.0" width="38.0" 
    319                             height="118.0" latexdraw:borderPos="outer" 
    320                             stroke-width="2.0" stroke="black" fill="#fafac9"/> 
    321                     </g> 
    322                 </g> 
    323                 <g latexdraw:type="square" id="id3338"> 
    324                     <rect x="811.0" y="121.0" width="18.0" height="18.0" 
    325                         latexdraw:borderPos="outer" stroke-width="2.0" 
    326                         stroke="black" fill="red"/> 
    327                 </g> 
    328                 <g latexdraw:type="square" id="id3339"> 
    329                     <rect x="811.0" y="161.0" width="18.0" height="18.0" 
    330                         latexdraw:borderPos="outer" stroke-width="2.0" 
    331                         stroke="black" fill="white"/> 
    332                 </g> 
    333                 <g latexdraw:type="square" id="id3340"> 
    334                     <rect x="811.0" y="201.0" width="18.0" height="18.0" 
    335                         latexdraw:borderPos="outer" stroke-width="2.0" 
    336                         stroke="black" fill="white"/> 
    337                 </g> 
    338                 <g latexdraw:type="square" id="id3341"> 
    339                     <rect x="811.0" y="261.0" width="18.0" height="18.0" 
    340                         latexdraw:borderPos="outer" stroke-width="2.0" 
    341                         stroke="black" fill="red"/> 
    342                 </g> 
    343                 <g latexdraw:type="square" id="id3342"> 
    344                     <rect x="811.0" y="301.0" width="18.0" height="18.0" 
    345                         latexdraw:borderPos="outer" stroke-width="2.0" 
    346                         stroke="black" fill="red"/> 
    347                 </g> 
    348                 <g latexdraw:type="square" id="id3343"> 
    349                     <rect x="811.0" y="341.0" width="18.0" height="18.0" 
    350                         latexdraw:borderPos="outer" stroke-width="2.0" 
    351                         stroke="black" fill="white"/> 
    352                 </g> 
    353                 <g latexdraw:type="square" id="id3344"> 
    354                     <rect x="811.0" y="401.0" width="18.0" height="18.0" 
    355                         latexdraw:borderPos="outer" stroke-width="2.0" 
    356                         stroke="black" fill="red"/> 
    357                 </g> 
    358                 <g latexdraw:type="square" id="id3345"> 
    359                     <rect x="811.0" y="441.0" width="18.0" height="18.0" 
    360                         latexdraw:borderPos="outer" stroke-width="2.0" 
    361                         stroke="black" fill="white"/> 
    362                 </g> 
    363                 <g latexdraw:type="square" id="id3346"> 
    364                     <rect x="811.0" y="481.0" width="18.0" height="18.0" 
    365                         latexdraw:borderPos="outer" stroke-width="2.0" 
    366                         stroke="black" fill="white"/> 
    367                 </g> 
    368                 <g latexdraw:type="square" id="id3347"> 
    369                     <rect x="811.0" y="541.0" width="18.0" height="18.0" 
    370                         latexdraw:borderPos="outer" stroke-width="2.0" 
    371                         stroke="black" fill="red"/> 
    372                 </g> 
    373                 <g latexdraw:type="square" id="id3348"> 
    374                     <rect x="811.0" y="581.0" width="18.0" height="18.0" 
    375                         latexdraw:borderPos="outer" stroke-width="2.0" 
    376                         stroke="black" fill="white"/> 
    377                 </g> 
    378                 <g latexdraw:type="square" id="id3349"> 
    379                     <rect x="811.0" y="621.0" width="18.0" height="18.0" 
    380                         latexdraw:borderPos="outer" stroke-width="2.0" 
    381                         stroke="black" fill="white"/> 
    382                 </g> 
    383                 <g latexdraw:type="text" id="id3350" 
    384                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    385                     <text x="770.0" y="140.0"><![CDATA[0]]></text> 
    386                 </g> 
    387                 <g latexdraw:type="text" id="id3353" 
    388                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    389                     <text x="770.0" y="180.0"><![CDATA[1]]></text> 
    390                 </g> 
    391                 <g latexdraw:type="text" id="id3356" 
    392                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    393                     <text x="770.0" y="220.0"><![CDATA[2]]></text> 
    394                 </g> 
    395                 <g latexdraw:type="text" id="id3359" 
    396                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    397                     <text x="770.0" y="280.0"><![CDATA[3]]></text> 
    398                 </g> 
    399                 <g latexdraw:type="text" id="id3362" 
    400                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    401                     <text x="770.0" y="320.0"><![CDATA[4]]></text> 
    402                 </g> 
    403                 <g latexdraw:type="text" id="id3365" 
    404                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    405                     <text x="770.0" y="360.0"><![CDATA[5]]></text> 
    406                 </g> 
    407                 <g latexdraw:type="text" id="id3368" 
    408                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    409                     <text x="770.0" y="420.0"><![CDATA[6]]></text> 
    410                 </g> 
    411                 <g latexdraw:type="text" id="id3371" 
    412                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    413                     <text x="770.0" y="460.0"><![CDATA[7]]></text> 
    414                 </g> 
    415                 <g latexdraw:type="text" id="id3374" 
    416                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    417                     <text x="770.0" y="500.0"><![CDATA[8]]></text> 
    418                 </g> 
    419                 <g latexdraw:type="text" id="id3377" 
    420                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    421                     <text x="770.0" y="560.0"><![CDATA[9]]></text> 
    422                 </g> 
    423                 <g latexdraw:type="text" id="id3380" 
    424                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    425                     <text x="770.0" y="600.0"><![CDATA[10]]></text> 
    426                 </g> 
    427                 <g latexdraw:type="text" id="id3383" 
    428                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 
    429                     <text x="770.0" y="640.0"><![CDATA[11]]></text> 
    430                 </g> 
    431                 <g latexdraw:type="line" id="id3386"> 
    432                     <line x1="640.0" x2="760.0" y1="380.0" y2="380.0" 
    433                         stroke-width="6.0" stroke="black" 
    434                         latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3386)"/> 
    435                 </g> 
    436                 <g latexdraw:type="text" id="id3387" 
    437                     font-family="Dialog" fill="black" font-size="18" latexdraw:x="670.0"> 
    438                     <text x="670.0" y="360.0"><![CDATA[step 2]]></text> 
    439                 </g> 
    440             </g> 
    441             <g latexdraw:type="line" id="id3391"> 
    442                 <line x1="410.0" x2="530.0" y1="380.0" y2="380.0" 
    443                     stroke-width="6.0" stroke="black" 
    444                     latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3391)"/> 
    445             </g> 
    446             <g latexdraw:type="text" id="id3392" font-family="Dialog" 
    447                 fill="black" font-size="18" latexdraw:x="440.0"> 
    448                 <text x="440.0" y="360.0"><![CDATA[step 1]]></text> 
    449             </g> 
    450             <g latexdraw:type="drawing" id="id3395"> 
    451                 <g latexdraw:type="rectangle" id="id3396"> 
    452                     <rect x="1011.0" y="531.0" width="38.0" 
     18        <g latexdraw:type="drawing" id="id528"> 
     19            <g latexdraw:type="drawing" id="id529"> 
     20                <g latexdraw:type="rectangle" id="id530"> 
     21                    <rect x="581.0" y="531.0" width="38.0" 
    45322                        height="118.0" latexdraw:borderPos="outer" 
    45423                        stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 
    45524                </g> 
    456                 <g latexdraw:type="rectangle" id="id3397"> 
    457                     <rect x="1011.0" y="391.0" width="38.0" 
     25                <g latexdraw:type="rectangle" id="id531"> 
     26                    <rect x="581.0" y="391.0" width="38.0" 
    45827                        height="118.0" latexdraw:borderPos="outer" 
    45928                        stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 
    46029                </g> 
    461                 <g latexdraw:type="rectangle" id="id3398"> 
    462                     <rect x="1011.0" y="251.0" width="38.0" 
     30                <g latexdraw:type="rectangle" id="id532"> 
     31                    <rect x="581.0" y="251.0" width="38.0" 
    46332                        height="118.0" latexdraw:borderPos="outer" 
    46433                        stroke-width="2.0" stroke="black" fill="#f1daf9"/> 
    46534                </g> 
    466                 <g latexdraw:type="rectangle" id="id3399"> 
    467                     <rect x="1011.0" y="111.0" width="38.0" 
     35                <g latexdraw:type="rectangle" id="id533"> 
     36                    <rect x="581.0" y="111.0" width="38.0" 
    46837                        height="118.0" latexdraw:borderPos="outer" 
    46938                        stroke-width="2.0" stroke="black" fill="#fafac9"/> 
    47039                </g> 
    47140            </g> 
    472             <g latexdraw:type="drawing" id="id3401"> 
    473                 <g latexdraw:type="square" id="id3402"> 
    474                     <rect x="1021.0" y="261.0" width="18.0" 
    475                         height="18.0" latexdraw:borderPos="outer" 
    476                         stroke-width="2.0" stroke="black" fill="red"/> 
    477                 </g> 
    478                 <g latexdraw:type="square" id="id3403"> 
    479                     <rect x="1021.0" y="301.0" width="18.0" 
    480                         height="18.0" latexdraw:borderPos="outer" 
    481                         stroke-width="2.0" stroke="black" fill="red"/> 
    482                 </g> 
    483                 <g latexdraw:type="square" id="id3404"> 
    484                     <rect x="1021.0" y="341.0" width="18.0" 
    485                         height="18.0" latexdraw:borderPos="outer" 
    486                         stroke-width="2.0" stroke="black" fill="red"/> 
    487                 </g> 
    488             </g> 
    489             <g latexdraw:type="text" id="id3406" font-family="Dialog" 
    490                 fill="black" font-size="18" latexdraw:x="980.0"> 
    491                 <text x="980.0" y="140.0"><![CDATA[0]]></text> 
    492             </g> 
    493             <g latexdraw:type="text" id="id3409" font-family="Dialog" 
    494                 fill="black" font-size="18" latexdraw:x="980.0"> 
    495                 <text x="980.0" y="180.0"><![CDATA[1]]></text> 
    496             </g> 
    497             <g latexdraw:type="text" id="id3412" font-family="Dialog" 
    498                 fill="black" font-size="18" latexdraw:x="980.0"> 
    499                 <text x="980.0" y="220.0"><![CDATA[2]]></text> 
    500             </g> 
    501             <g latexdraw:type="text" id="id3415" font-family="Dialog" 
    502                 fill="black" font-size="18" latexdraw:x="980.0"> 
    503                 <text x="980.0" y="280.0"><![CDATA[3]]></text> 
    504             </g> 
    505             <g latexdraw:type="text" id="id3418" font-family="Dialog" 
    506                 fill="black" font-size="18" latexdraw:x="980.0"> 
    507                 <text x="980.0" y="320.0"><![CDATA[4]]></text> 
    508             </g> 
    509             <g latexdraw:type="text" id="id3421" font-family="Dialog" 
    510                 fill="black" font-size="18" latexdraw:x="980.0"> 
    511                 <text x="980.0" y="360.0"><![CDATA[5]]></text> 
    512             </g> 
    513             <g latexdraw:type="text" id="id3424" font-family="Dialog" 
    514                 fill="black" font-size="18" latexdraw:x="980.0"> 
    515                 <text x="980.0" y="420.0"><![CDATA[6]]></text> 
    516             </g> 
    517             <g latexdraw:type="text" id="id3427" font-family="Dialog" 
    518                 fill="black" font-size="18" latexdraw:x="980.0"> 
    519                 <text x="980.0" y="460.0"><![CDATA[7]]></text> 
    520             </g> 
    521             <g latexdraw:type="text" id="id3430" font-family="Dialog" 
    522                 fill="black" font-size="18" latexdraw:x="980.0"> 
    523                 <text x="980.0" y="500.0"><![CDATA[8]]></text> 
    524             </g> 
    525             <g latexdraw:type="text" id="id3433" font-family="Dialog" 
    526                 fill="black" font-size="18" latexdraw:x="980.0"> 
    527                 <text x="980.0" y="560.0"><![CDATA[9]]></text> 
    528             </g> 
    529             <g latexdraw:type="text" id="id3436" font-family="Dialog" 
    530                 fill="black" font-size="18" latexdraw:x="980.0"> 
    531                 <text x="980.0" y="600.0"><![CDATA[10]]></text> 
    532             </g> 
    533             <g latexdraw:type="text" id="id3439" font-family="Dialog" 
    534                 fill="black" font-size="18" latexdraw:x="980.0"> 
    535                 <text x="980.0" y="640.0"><![CDATA[11]]></text> 
    536             </g> 
    537             <g latexdraw:type="line" id="id3442"> 
    538                 <line x1="850.0" x2="970.0" y1="380.0" y2="380.0" 
    539                     stroke-width="6.0" stroke="black" 
    540                     latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3442)"/> 
    541             </g> 
    542             <g latexdraw:type="text" id="id3443" font-family="Dialog" 
    543                 fill="black" font-size="18" latexdraw:x="880.0"> 
    544                 <text x="880.0" y="360.0"><![CDATA[step 3]]></text> 
    545             </g> 
    546             <g latexdraw:type="drawing" id="id3446"> 
    547                 <g latexdraw:type="square" id="id3447"> 
    548                     <rect x="1021.0" y="401.0" width="18.0" 
    549                         height="18.0" latexdraw:borderPos="outer" 
    550                         stroke-width="2.0" stroke="black" fill="red"/> 
    551                 </g> 
    552                 <g latexdraw:type="square" id="id3448"> 
    553                     <rect x="1021.0" y="441.0" width="18.0" 
    554                         height="18.0" latexdraw:borderPos="outer" 
    555                         stroke-width="2.0" stroke="black" fill="red"/> 
    556                 </g> 
    557                 <g latexdraw:type="square" id="id3449"> 
    558                     <rect x="1021.0" y="481.0" width="18.0" 
    559                         height="18.0" latexdraw:borderPos="outer" 
    560                         stroke-width="2.0" stroke="black" fill="red"/> 
    561                 </g> 
    562             </g> 
    563             <g latexdraw:type="drawing" id="id3451"> 
    564                 <g latexdraw:type="square" id="id3452"> 
    565                     <rect x="1021.0" y="541.0" width="18.0" 
    566                         height="18.0" latexdraw:borderPos="outer" 
    567                         stroke-width="2.0" stroke="black" fill="red"/> 
    568                 </g> 
    569                 <g latexdraw:type="square" id="id3453"> 
    570                     <rect x="1021.0" y="581.0" width="18.0" 
    571                         height="18.0" latexdraw:borderPos="outer" 
    572                         stroke-width="2.0" stroke="black" fill="red"/> 
    573                 </g> 
    574                 <g latexdraw:type="square" id="id3454"> 
    575                     <rect x="1021.0" y="621.0" width="18.0" 
    576                         height="18.0" latexdraw:borderPos="outer" 
    577                         stroke-width="2.0" stroke="black" fill="red"/> 
    578                 </g> 
    579             </g> 
    580             <g latexdraw:type="drawing" id="id3456"> 
    581                 <g latexdraw:type="square" id="id3457"> 
    582                     <rect x="1021.0" y="121.0" width="18.0" 
    583                         height="18.0" latexdraw:borderPos="outer" 
    584                         stroke-width="2.0" stroke="black" fill="red"/> 
    585                 </g> 
    586                 <g latexdraw:type="square" id="id3458"> 
    587                     <rect x="1021.0" y="161.0" width="18.0" 
    588                         height="18.0" latexdraw:borderPos="outer" 
    589                         stroke-width="2.0" stroke="black" fill="red"/> 
    590                 </g> 
    591                 <g latexdraw:type="square" id="id3459"> 
    592                     <rect x="1021.0" y="201.0" width="18.0" 
    593                         height="18.0" latexdraw:borderPos="outer" 
    594                         stroke-width="2.0" stroke="black" fill="red"/> 
    595                 </g> 
    596             </g> 
     41            <g latexdraw:type="drawing" id="id535"> 
     42                <g latexdraw:type="square" id="id536"> 
     43                    <rect x="591.0" y="121.0" width="18.0" height="18.0" 
     44                        latexdraw:borderPos="outer" stroke-width="2.0" 
     45                        stroke="black" fill="white"/> 
     46                </g> 
     47                <g latexdraw:type="square" id="id537"> 
     48                    <rect x="591.0" y="161.0" width="18.0" height="18.0" 
     49                        latexdraw:borderPos="outer" stroke-width="2.0" 
     50                        stroke="black" fill="white"/> 
     51                </g> 
     52                <g latexdraw:type="square" id="id538"> 
     53                    <rect x="591.0" y="201.0" width="18.0" height="18.0" 
     54                        latexdraw:borderPos="outer" stroke-width="2.0" 
     55                        stroke="black" fill="white"/> 
     56                </g> 
     57            </g> 
     58            <g latexdraw:type="square" id="id540"> 
     59                <rect x="591.0" y="261.0" width="18.0" height="18.0" 
     60                    latexdraw:borderPos="outer" stroke-width="2.0" 
     61                    stroke="black" fill="white"/> 
     62            </g> 
     63            <g latexdraw:type="square" id="id541"> 
     64                <rect x="591.0" y="301.0" width="18.0" height="18.0" 
     65                    latexdraw:borderPos="outer" stroke-width="2.0" 
     66                    stroke="black" fill="red"/> 
     67            </g> 
     68            <g latexdraw:type="square" id="id542"> 
     69                <rect x="591.0" y="341.0" width="18.0" height="18.0" 
     70                    latexdraw:borderPos="outer" stroke-width="2.0" 
     71                    stroke="black" fill="white"/> 
     72            </g> 
     73            <g latexdraw:type="drawing" id="id543"> 
     74                <g latexdraw:type="square" id="id544"> 
     75                    <rect x="591.0" y="401.0" width="18.0" height="18.0" 
     76                        latexdraw:borderPos="outer" stroke-width="2.0" 
     77                        stroke="black" fill="white"/> 
     78                </g> 
     79                <g latexdraw:type="square" id="id545"> 
     80                    <rect x="591.0" y="441.0" width="18.0" height="18.0" 
     81                        latexdraw:borderPos="outer" stroke-width="2.0" 
     82                        stroke="black" fill="white"/> 
     83                </g> 
     84                <g latexdraw:type="square" id="id546"> 
     85                    <rect x="591.0" y="481.0" width="18.0" height="18.0" 
     86                        latexdraw:borderPos="outer" stroke-width="2.0" 
     87                        stroke="black" fill="white"/> 
     88                </g> 
     89            </g> 
     90            <g latexdraw:type="drawing" id="id548"> 
     91                <g latexdraw:type="square" id="id549"> 
     92                    <rect x="591.0" y="541.0" width="18.0" height="18.0" 
     93                        latexdraw:borderPos="outer" stroke-width="2.0" 
     94                        stroke="black" fill="white"/> 
     95                </g> 
     96                <g latexdraw:type="square" id="id550"> 
     97                    <rect x="591.0" y="581.0" width="18.0" height="18.0" 
     98                        latexdraw:borderPos="outer" stroke-width="2.0" 
     99                        stroke="black" fill="white"/> 
     100                </g> 
     101                <g latexdraw:type="square" id="id551"> 
     102                    <rect x="591.0" y="621.0" width="18.0" height="18.0" 
     103                        latexdraw:borderPos="outer" stroke-width="2.0" 
     104                        stroke="black" fill="white"/> 
     105                </g> 
     106            </g> 
     107            <g latexdraw:type="text" id="id553" font-family="Dialog" 
     108                fill="black" font-size="18" latexdraw:x="550.0"> 
     109                <text x="550.0" y="140.0"><![CDATA[0]]></text> 
     110            </g> 
     111            <g latexdraw:type="text" id="id555" font-family="Dialog" 
     112                fill="black" font-size="18" latexdraw:x="550.0"> 
     113                <text x="550.0" y="180.0"><![CDATA[1]]></text> 
     114            </g> 
     115            <g latexdraw:type="text" id="id557" font-family="Dialog" 
     116                fill="black" font-size="18" latexdraw:x="550.0"> 
     117                <text x="550.0" y="220.0"><![CDATA[2]]></text> 
     118            </g> 
     119            <g latexdraw:type="text" id="id559" font-family="Dialog" 
     120                fill="black" font-size="18" latexdraw:x="550.0"> 
     121                <text x="550.0" y="280.0"><![CDATA[3]]></text> 
     122            </g> 
     123            <g latexdraw:type="text" id="id561" font-family="Dialog" 
     124                fill="black" font-size="18" latexdraw:x="550.0"> 
     125                <text x="550.0" y="320.0"><![CDATA[4]]></text> 
     126            </g> 
     127            <g latexdraw:type="text" id="id563" font-family="Dialog" 
     128                fill="black" font-size="18" latexdraw:x="550.0"> 
     129                <text x="550.0" y="360.0"><![CDATA[5]]></text> 
     130            </g> 
     131            <g latexdraw:type="text" id="id565" font-family="Dialog" 
     132                fill="black" font-size="18" latexdraw:x="550.0"> 
     133                <text x="550.0" y="420.0"><![CDATA[6]]></text> 
     134            </g> 
     135            <g latexdraw:type="text" id="id567" font-family="Dialog" 
     136                fill="black" font-size="18" latexdraw:x="550.0"> 
     137                <text x="550.0" y="460.0"><![CDATA[7]]></text> 
     138            </g> 
     139            <g latexdraw:type="text" id="id569" font-family="Dialog" 
     140                fill="black" font-size="18" latexdraw:x="550.0"> 
     141                <text x="550.0" y="500.0"><![CDATA[8]]></text> 
     142            </g> 
     143            <g latexdraw:type="text" id="id571" font-family="Dialog" 
     144                fill="black" font-size="18" latexdraw:x="550.0"> 
     145                <text x="550.0" y="560.0"><![CDATA[9]]></text> 
     146            </g> 
     147            <g latexdraw:type="text" id="id573" font-family="Dialog" 
     148                fill="black" font-size="18" latexdraw:x="550.0"> 
     149                <text x="550.0" y="600.0"><![CDATA[10]]></text> 
     150            </g> 
     151            <g latexdraw:type="text" id="id575" font-family="Dialog" 
     152                fill="black" font-size="18" latexdraw:x="550.0"> 
     153                <text x="550.0" y="640.0"><![CDATA[11]]></text> 
     154            </g> 
     155        </g> 
     156        <g latexdraw:type="drawing" id="id578"> 
     157            <g latexdraw:type="rectangle" id="id579"> 
     158                <rect x="801.0" y="531.0" width="38.0" height="118.0" 
     159                    latexdraw:borderPos="outer" stroke-width="2.0" 
     160                    stroke="black" fill="#d4f4e0"/> 
     161            </g> 
     162            <g latexdraw:type="rectangle" id="id580"> 
     163                <rect x="801.0" y="391.0" width="38.0" height="118.0" 
     164                    latexdraw:borderPos="outer" stroke-width="2.0" 
     165                    stroke="black" fill="#dfe1f2"/> 
     166            </g> 
     167            <g latexdraw:type="rectangle" id="id581"> 
     168                <rect x="801.0" y="251.0" width="38.0" height="118.0" 
     169                    latexdraw:borderPos="outer" stroke-width="2.0" 
     170                    stroke="black" fill="#f1daf9"/> 
     171            </g> 
     172            <g latexdraw:type="rectangle" id="id582"> 
     173                <rect x="801.0" y="111.0" width="38.0" height="118.0" 
     174                    latexdraw:borderPos="outer" stroke-width="2.0" 
     175                    stroke="black" fill="#fafac9"/> 
     176            </g> 
     177        </g> 
     178        <g latexdraw:type="square" id="id584"> 
     179            <rect x="811.0" y="121.0" width="18.0" height="18.0" 
     180                latexdraw:borderPos="outer" stroke-width="2.0" 
     181                stroke="black" fill="red"/> 
     182        </g> 
     183        <g latexdraw:type="square" id="id585"> 
     184            <rect x="811.0" y="161.0" width="18.0" height="18.0" 
     185                latexdraw:borderPos="outer" stroke-width="2.0" 
     186                stroke="black" fill="white"/> 
     187        </g> 
     188        <g latexdraw:type="square" id="id586"> 
     189            <rect x="811.0" y="201.0" width="18.0" height="18.0" 
     190                latexdraw:borderPos="outer" stroke-width="2.0" 
     191                stroke="black" fill="white"/> 
     192        </g> 
     193        <g latexdraw:type="square" id="id587"> 
     194            <rect x="811.0" y="301.0" width="18.0" height="18.0" 
     195                latexdraw:borderPos="outer" stroke-width="2.0" 
     196                stroke="black" fill="red"/> 
     197        </g> 
     198        <g latexdraw:type="square" id="id588"> 
     199            <rect x="811.0" y="341.0" width="18.0" height="18.0" 
     200                latexdraw:borderPos="outer" stroke-width="2.0" 
     201                stroke="black" fill="white"/> 
     202        </g> 
     203        <g latexdraw:type="square" id="id589"> 
     204            <rect x="811.0" y="401.0" width="18.0" height="18.0" 
     205                latexdraw:borderPos="outer" stroke-width="2.0" 
     206                stroke="black" fill="red"/> 
     207        </g> 
     208        <g latexdraw:type="square" id="id590"> 
     209            <rect x="811.0" y="441.0" width="18.0" height="18.0" 
     210                latexdraw:borderPos="outer" stroke-width="2.0" 
     211                stroke="black" fill="white"/> 
     212        </g> 
     213        <g latexdraw:type="square" id="id591"> 
     214            <rect x="811.0" y="481.0" width="18.0" height="18.0" 
     215                latexdraw:borderPos="outer" stroke-width="2.0" 
     216                stroke="black" fill="white"/> 
     217        </g> 
     218        <g latexdraw:type="square" id="id592"> 
     219            <rect x="811.0" y="541.0" width="18.0" height="18.0" 
     220                latexdraw:borderPos="outer" stroke-width="2.0" 
     221                stroke="black" fill="red"/> 
     222        </g> 
     223        <g latexdraw:type="square" id="id593"> 
     224            <rect x="811.0" y="581.0" width="18.0" height="18.0" 
     225                latexdraw:borderPos="outer" stroke-width="2.0" 
     226                stroke="black" fill="white"/> 
     227        </g> 
     228        <g latexdraw:type="square" id="id594"> 
     229            <rect x="811.0" y="621.0" width="18.0" height="18.0" 
     230                latexdraw:borderPos="outer" stroke-width="2.0" 
     231                stroke="black" fill="white"/> 
     232        </g> 
     233        <g latexdraw:type="text" id="id595" font-family="Dialog" 
     234            fill="black" font-size="18" latexdraw:x="770.0"> 
     235            <text x="770.0" y="140.0"><![CDATA[0]]></text> 
     236        </g> 
     237        <g latexdraw:type="text" id="id597" font-family="Dialog" 
     238            fill="black" font-size="18" latexdraw:x="770.0"> 
     239            <text x="770.0" y="180.0"><![CDATA[1]]></text> 
     240        </g> 
     241        <g latexdraw:type="text" id="id599" font-family="Dialog" 
     242            fill="black" font-size="18" latexdraw:x="770.0"> 
     243            <text x="770.0" y="220.0"><![CDATA[2]]></text> 
     244        </g> 
     245        <g latexdraw:type="text" id="id601" font-family="Dialog" 
     246            fill="black" font-size="18" latexdraw:x="770.0"> 
     247            <text x="770.0" y="280.0"><![CDATA[3]]></text> 
     248        </g> 
     249        <g latexdraw:type="text" id="id603" font-family="Dialog" 
     250            fill="black" font-size="18" latexdraw:x="770.0"> 
     251            <text x="770.0" y="320.0"><![CDATA[4]]></text> 
     252        </g> 
     253        <g latexdraw:type="text" id="id605" font-family="Dialog" 
     254            fill="black" font-size="18" latexdraw:x="770.0"> 
     255            <text x="770.0" y="360.0"><![CDATA[5]]></text> 
     256        </g> 
     257        <g latexdraw:type="text" id="id607" font-family="Dialog" 
     258            fill="black" font-size="18" latexdraw:x="770.0"> 
     259            <text x="770.0" y="420.0"><![CDATA[6]]></text> 
     260        </g> 
     261        <g latexdraw:type="text" id="id609" font-family="Dialog" 
     262            fill="black" font-size="18" latexdraw:x="770.0"> 
     263            <text x="770.0" y="460.0"><![CDATA[7]]></text> 
     264        </g> 
     265        <g latexdraw:type="text" id="id611" font-family="Dialog" 
     266            fill="black" font-size="18" latexdraw:x="770.0"> 
     267            <text x="770.0" y="500.0"><![CDATA[8]]></text> 
     268        </g> 
     269        <g latexdraw:type="text" id="id613" font-family="Dialog" 
     270            fill="black" font-size="18" latexdraw:x="770.0"> 
     271            <text x="770.0" y="560.0"><![CDATA[9]]></text> 
     272        </g> 
     273        <g latexdraw:type="text" id="id615" font-family="Dialog" 
     274            fill="black" font-size="18" latexdraw:x="770.0"> 
     275            <text x="770.0" y="600.0"><![CDATA[10]]></text> 
     276        </g> 
     277        <g latexdraw:type="text" id="id617" font-family="Dialog" 
     278            fill="black" font-size="18" latexdraw:x="770.0"> 
     279            <text x="770.0" y="640.0"><![CDATA[11]]></text> 
     280        </g> 
     281        <g latexdraw:type="line" id="id619"> 
     282            <line x1="640.0" x2="760.0" y1="380.0" y2="380.0" 
     283                stroke-width="6.0" stroke="black" 
     284                latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-619)"/> 
     285        </g> 
     286        <g latexdraw:type="text" id="id620" font-family="Dialog" 
     287            fill="black" font-size="18" latexdraw:x="670.0"> 
     288            <text x="670.0" y="360.0"><![CDATA[step 1]]></text> 
     289        </g> 
     290        <g latexdraw:type="drawing" id="id622"> 
     291            <g latexdraw:type="rectangle" id="id623"> 
     292                <rect x="1011.0" y="531.0" width="38.0" height="118.0" 
     293                    latexdraw:borderPos="outer" stroke-width="2.0" 
     294                    stroke="black" fill="#d4f4e0"/> 
     295            </g> 
     296            <g latexdraw:type="rectangle" id="id624"> 
     297                <rect x="1011.0" y="391.0" width="38.0" height="118.0" 
     298                    latexdraw:borderPos="outer" stroke-width="2.0" 
     299                    stroke="black" fill="#dfe1f2"/> 
     300            </g> 
     301            <g latexdraw:type="rectangle" id="id625"> 
     302                <rect x="1011.0" y="251.0" width="38.0" height="118.0" 
     303                    latexdraw:borderPos="outer" stroke-width="2.0" 
     304                    stroke="black" fill="#f1daf9"/> 
     305            </g> 
     306            <g latexdraw:type="rectangle" id="id626"> 
     307                <rect x="1011.0" y="111.0" width="38.0" height="118.0" 
     308                    latexdraw:borderPos="outer" stroke-width="2.0" 
     309                    stroke="black" fill="#fafac9"/> 
     310            </g> 
     311        </g> 
     312        <g latexdraw:type="drawing" id="id628"> 
     313            <g latexdraw:type="square" id="id629"> 
     314                <rect x="1021.0" y="261.0" width="18.0" height="18.0" 
     315                    latexdraw:borderPos="outer" stroke-width="2.0" 
     316                    stroke="black" fill="red"/> 
     317            </g> 
     318            <g latexdraw:type="square" id="id630"> 
     319                <rect x="1021.0" y="301.0" width="18.0" height="18.0" 
     320                    latexdraw:borderPos="outer" stroke-width="2.0" 
     321                    stroke="black" fill="red"/> 
     322            </g> 
     323            <g latexdraw:type="square" id="id631"> 
     324                <rect x="1021.0" y="341.0" width="18.0" height="18.0" 
     325                    latexdraw:borderPos="outer" stroke-width="2.0" 
     326                    stroke="black" fill="red"/> 
     327            </g> 
     328        </g> 
     329        <g latexdraw:type="text" id="id633" font-family="Dialog" 
     330            fill="black" font-size="18" latexdraw:x="980.0"> 
     331            <text x="980.0" y="140.0"><![CDATA[0]]></text> 
     332        </g> 
     333        <g latexdraw:type="text" id="id635" font-family="Dialog" 
     334            fill="black" font-size="18" latexdraw:x="980.0"> 
     335            <text x="980.0" y="180.0"><![CDATA[1]]></text> 
     336        </g> 
     337        <g latexdraw:type="text" id="id637" font-family="Dialog" 
     338            fill="black" font-size="18" latexdraw:x="980.0"> 
     339            <text x="980.0" y="220.0"><![CDATA[2]]></text> 
     340        </g> 
     341        <g latexdraw:type="text" id="id639" font-family="Dialog" 
     342            fill="black" font-size="18" latexdraw:x="980.0"> 
     343            <text x="980.0" y="280.0"><![CDATA[3]]></text> 
     344        </g> 
     345        <g latexdraw:type="text" id="id641" font-family="Dialog" 
     346            fill="black" font-size="18" latexdraw:x="980.0"> 
     347            <text x="980.0" y="320.0"><![CDATA[4]]></text> 
     348        </g> 
     349        <g latexdraw:type="text" id="id643" font-family="Dialog" 
     350            fill="black" font-size="18" latexdraw:x="980.0"> 
     351            <text x="980.0" y="360.0"><![CDATA[5]]></text> 
     352        </g> 
     353        <g latexdraw:type="text" id="id645" font-family="Dialog" 
     354            fill="black" font-size="18" latexdraw:x="980.0"> 
     355            <text x="980.0" y="420.0"><![CDATA[6]]></text> 
     356        </g> 
     357        <g latexdraw:type="text" id="id647" font-family="Dialog" 
     358            fill="black" font-size="18" latexdraw:x="980.0"> 
     359            <text x="980.0" y="460.0"><![CDATA[7]]></text> 
     360        </g> 
     361        <g latexdraw:type="text" id="id649" font-family="Dialog" 
     362            fill="black" font-size="18" latexdraw:x="980.0"> 
     363            <text x="980.0" y="500.0"><![CDATA[8]]></text> 
     364        </g> 
     365        <g latexdraw:type="text" id="id651" font-family="Dialog" 
     366            fill="black" font-size="18" latexdraw:x="980.0"> 
     367            <text x="980.0" y="560.0"><![CDATA[9]]></text> 
     368        </g> 
     369        <g latexdraw:type="text" id="id653" font-family="Dialog" 
     370            fill="black" font-size="18" latexdraw:x="980.0"> 
     371            <text x="980.0" y="600.0"><![CDATA[10]]></text> 
     372        </g> 
     373        <g latexdraw:type="text" id="id655" font-family="Dialog" 
     374            fill="black" font-size="18" latexdraw:x="980.0"> 
     375            <text x="980.0" y="640.0"><![CDATA[11]]></text> 
     376        </g> 
     377        <g latexdraw:type="line" id="id657"> 
     378            <line x1="850.0" x2="970.0" y1="380.0" y2="380.0" 
     379                stroke-width="6.0" stroke="black" 
     380                latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-657)"/> 
     381        </g> 
     382        <g latexdraw:type="text" id="id658" font-family="Dialog" 
     383            fill="black" font-size="18" latexdraw:x="880.0"> 
     384            <text x="880.0" y="360.0"><![CDATA[step2]]></text> 
     385        </g> 
     386        <g latexdraw:type="drawing" id="id661"> 
     387            <g latexdraw:type="square" id="id662"> 
     388                <rect x="1021.0" y="401.0" width="18.0" height="18.0" 
     389                    latexdraw:borderPos="outer" stroke-width="2.0" 
     390                    stroke="black" fill="red"/> 
     391            </g> 
     392            <g latexdraw:type="square" id="id663"> 
     393                <rect x="1021.0" y="441.0" width="18.0" height="18.0" 
     394                    latexdraw:borderPos="outer" stroke-width="2.0" 
     395                    stroke="black" fill="red"/> 
     396            </g> 
     397            <g latexdraw:type="square" id="id664"> 
     398                <rect x="1021.0" y="481.0" width="18.0" height="18.0" 
     399                    latexdraw:borderPos="outer" stroke-width="2.0" 
     400                    stroke="black" fill="red"/> 
     401            </g> 
     402        </g> 
     403        <g latexdraw:type="drawing" id="id666"> 
     404            <g latexdraw:type="square" id="id667"> 
     405                <rect x="1021.0" y="541.0" width="18.0" height="18.0" 
     406                    latexdraw:borderPos="outer" stroke-width="2.0" 
     407                    stroke="black" fill="red"/> 
     408            </g> 
     409            <g latexdraw:type="square" id="id668"> 
     410                <rect x="1021.0" y="581.0" width="18.0" height="18.0" 
     411                    latexdraw:borderPos="outer" stroke-width="2.0" 
     412                    stroke="black" fill="red"/> 
     413            </g> 
     414            <g latexdraw:type="square" id="id669"> 
     415                <rect x="1021.0" y="621.0" width="18.0" height="18.0" 
     416                    latexdraw:borderPos="outer" stroke-width="2.0" 
     417                    stroke="black" fill="red"/> 
     418            </g> 
     419        </g> 
     420        <g latexdraw:type="drawing" id="id671"> 
     421            <g latexdraw:type="square" id="id672"> 
     422                <rect x="1021.0" y="121.0" width="18.0" height="18.0" 
     423                    latexdraw:borderPos="outer" stroke-width="2.0" 
     424                    stroke="black" fill="red"/> 
     425            </g> 
     426            <g latexdraw:type="square" id="id673"> 
     427                <rect x="1021.0" y="161.0" width="18.0" height="18.0" 
     428                    latexdraw:borderPos="outer" stroke-width="2.0" 
     429                    stroke="black" fill="red"/> 
     430            </g> 
     431            <g latexdraw:type="square" id="id674"> 
     432                <rect x="1021.0" y="201.0" width="18.0" height="18.0" 
     433                    latexdraw:borderPos="outer" stroke-width="2.0" 
     434                    stroke="black" fill="red"/> 
     435            </g> 
     436        </g> 
     437        <g latexdraw:type="square" id="id676"> 
     438            <rect x="811.0" y="261.0" width="18.0" height="18.0" 
     439                latexdraw:borderPos="outer" stroke-width="2.0" 
     440                stroke="black" fill="white"/> 
    597441        </g> 
    598442    </g> 
     
    612456            <latexdraw:version>2.0.8</latexdraw:version> 
    613457            <latexdraw:size> 
    614                 <latexdraw:width>1867</latexdraw:width> 
    615                 <latexdraw:height>1176</latexdraw:height> 
     458                <latexdraw:width>2507</latexdraw:width> 
     459                <latexdraw:height>1576</latexdraw:height> 
    616460            </latexdraw:size> 
    617461            <latexdraw:position> 
     
    619463                <latexdraw:y>24</latexdraw:y> 
    620464            </latexdraw:position> 
    621             <latexdraw:dividerPosition>0.8489555436529191</latexdraw:dividerPosition> 
     465            <latexdraw:dividerPosition>0.9976067012365377</latexdraw:dividerPosition> 
    622466            <latexdraw:displayCodePanel>true</latexdraw:displayCodePanel> 
    623467            <latexdraw:displayXScale>true</latexdraw:displayXScale> 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.aux

    r1551 r1552  
    88\@writefile{lof}{\contentsline {figure}{\numberline {3}{\ignorespaces }}{3}} 
    99\newlabel{fig:omp}{{3}{3}} 
    10 \@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces }}{3}} 
    11 \newlabel{fig:scheme}{{4}{3}} 
    12 \@writefile{toc}{\contentsline {section}{\numberline {3}Performance of LMDZ using EP\_XIOS}{5}} 
    13 \@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces Speedup obtained by using EP in LMDZ simulations.}}{5}} 
     10\@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces }}{4}} 
     11\newlabel{fig:scheme}{{4}{4}} 
     12\@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces }}{5}} 
     13\newlabel{fig:tag}{{5}{5}} 
     14\@writefile{lof}{\contentsline {figure}{\numberline {6}{\ignorespaces This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1 and tag=65537. From the receiver's point of view, the endpoint 3 is actually receiving a message from MPI rank 0 with tag=65537.}}{5}} 
     15\newlabel{fig:sendrecv}{{6}{5}} 
     16\citation{ep:2018} 
     17\@writefile{lof}{\contentsline {figure}{\numberline {7}{\ignorespaces }}{6}} 
     18\newlabel{fig:bcast}{{7}{6}} 
     19\@writefile{lof}{\contentsline {figure}{\numberline {8}{\ignorespaces }}{6}} 
     20\newlabel{fig:allreduce}{{8}{6}} 
     21\citation{ep:2018} 
    1422\bibstyle{plain} 
    1523\bibdata{reference} 
    16 \bibcite{Dinan:2013}{1} 
    17 \bibcite{Sridharan:2014}{2} 
    18 \@writefile{toc}{\contentsline {section}{\numberline {4}Performance of EP\_XIOS}{6}} 
    19 \@writefile{toc}{\contentsline {section}{\numberline {5}Perspectives of EP\_XIOS}{6}} 
     24\bibcite{ep:2018}{1} 
     25\bibcite{Dinan:2013}{2} 
     26\bibcite{Sridharan:2014}{3} 
     27\@writefile{toc}{\contentsline {section}{\numberline {3}The multi-threaded XIOS and performance results}{7}} 
     28\@writefile{toc}{\contentsline {section}{\numberline {4}Future works for XIOS}{7}} 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.bbl

    r1551 r1552  
    11\begin{thebibliography}{1} 
     2 
     3\bibitem{ep:2018} 
     4XIOS developper group. 
     5\newblock Note for {XIOS} {E}ndpoints. 
     6\newblock Technical report, 
     7  \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp/Note}, 
     8  2018. 
    29 
    310\bibitem{Dinan:2013} 
    411J.~Dinan, Pavan Balaji, D.~Goodell, D.~Miller, M.~Snir, and Rajeev Thakur. 
    5 \newblock Enabling mpi interoperability through flexible communication 
    6   endpoints. 
     12\newblock Enabling {MPI} {I}nteroperability {T}hrough {F}lexible 
     13  {C}ommunication {E}ndpoints. 
    714\newblock In {\em EuroMPI 2013}, Madrid, Spain, 2013. 
    815 
    916\bibitem{Sridharan:2014} 
    1017S.~Sridharan, J.~Dinan, and D.~D. Kalamkar. 
    11 \newblock Enabling efficient multithreaded mpi communication through a 
    12   library-based implementation of mpi endpoints. 
     18\newblock Enabling {E}fficient {M}ultithreaded {MPI} {C}ommunication {T}hrough 
     19  a {L}ibrary-{B}ased {I}mplementation of {MPI} {E}ndpoints. 
    1320\newblock In {\em SC14: International Conference for High Performance 
    1421  Computing, Networking, Storage and Analysis}, pages 487--498, Nov 2014. 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.blg

    r1551 r1552  
    1 This is BibTeX, Version 0.99d (TeX Live 2016) 
     1This is BibTeX, Version 0.99d (TeX Live 2015/Debian) 
    22Capacity: max_strings=35307, hash_size=35307, hash_prime=30011 
    33The top-level auxiliary file: rapport ESIWACE.aux 
    44The style file: plain.bst 
    55Database file #1: reference.bib 
    6 You've used 2 entries, 
     6Warning--empty institution in ep:2018 
     7You've used 3 entries, 
    78            2118 wiz_defined-function locations, 
    8             512 strings with 4502 characters, 
    9 and the built_in function-call counts, 893 in all, are: 
    10 = -- 80 
    11 > -- 51 
     9            517 strings with 4672 characters, 
     10and the built_in function-call counts, 1094 in all, are: 
     11= -- 101 
     12> -- 58 
    1213< -- 1 
    13 + -- 20 
    14 - -- 18 
    15 * -- 63 
    16 := -- 146 
    17 add.period$ -- 6 
    18 call.type$ -- 2 
    19 change.case$ -- 15 
     14+ -- 23 
     15- -- 20 
     16* -- 73 
     17:= -- 186 
     18add.period$ -- 9 
     19call.type$ -- 3 
     20change.case$ -- 20 
    2021chr.to.int$ -- 0 
    21 cite$ -- 2 
    22 duplicate$ -- 37 
    23 empty$ -- 68 
    24 format.name$ -- 18 
    25 if$ -- 192 
     22cite$ -- 4 
     23duplicate$ -- 45 
     24empty$ -- 83 
     25format.name$ -- 20 
     26if$ -- 231 
    2627int.to.chr$ -- 0 
    27 int.to.str$ -- 2 
     28int.to.str$ -- 3 
    2829missing$ -- 2 
    29 newline$ -- 13 
    30 num.names$ -- 4 
    31 pop$ -- 22 
     30newline$ -- 18 
     31num.names$ -- 6 
     32pop$ -- 25 
    3233preamble$ -- 1 
    33 purify$ -- 13 
     34purify$ -- 16 
    3435quote$ -- 0 
    35 skip$ -- 25 
     36skip$ -- 30 
    3637stack$ -- 0 
    37 substring$ -- 38 
     38substring$ -- 43 
    3839swap$ -- 12 
    3940text.length$ -- 1 
    4041text.prefix$ -- 0 
    4142top$ -- 0 
    42 type$ -- 8 
    43 warning$ -- 0 
    44 while$ -- 6 
    45 width$ -- 3 
    46 write$ -- 24 
     43type$ -- 12 
     44warning$ -- 1 
     45while$ -- 8 
     46width$ -- 4 
     47write$ -- 35 
     48(There was 1 warning) 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.log

    r1551 r1552  
    1 This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) (preloaded format=pdflatex 2016.7.25)  25 JUN 2018 17:20 
     1This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24)  27 JUN 2018 15:10 
    22entering extended mode 
    33 restricted \write18 enabled. 
     
    55**"rapport ESIWACE.tex" 
    66(./rapport ESIWACE.tex 
    7 LaTeX2e <2016/03/31> 
    8 Babel <3.9r> and hyphenation patterns for 83 language(s) loaded. 
    9 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/article.cls 
     7LaTeX2e <2016/02/01> 
     8Babel <3.9q> and hyphenation patterns for 3 language(s) loaded. 
     9(/usr/share/texlive/texmf-dist/tex/latex/base/article.cls 
    1010Document Class: article 2014/09/29 v1.4h Standard LaTeX document class 
    11 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/size10.clo 
     11(/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo 
    1212File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option) 
    1313) 
     
    2424\bibindent=\dimen102 
    2525) 
    26 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/inputenc.sty 
     26(/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty 
    2727Package: inputenc 2015/03/17 v1.2c Input encoding file 
    2828\inpenc@prehook=\toks14 
    2929\inpenc@posthook=\toks15 
    3030 
    31 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/utf8.def 
    32 File: utf8.def 2016/02/28 v1.1s UTF-8 support for inputenc 
     31(/usr/share/texlive/texmf-dist/tex/latex/base/utf8.def 
     32File: utf8.def 2015/12/03 v1.1r UTF-8 support for inputenc 
    3333Now handling font encoding OML ... 
    3434... no UTF-8 mapping file for font encoding OML 
     
    3636... processing UTF-8 mapping file for font encoding T1 
    3737 
    38 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/t1enc.dfu 
    39 File: t1enc.dfu 2016/02/28 v1.1s UTF-8 support for inputenc 
     38(/usr/share/texlive/texmf-dist/tex/latex/base/t1enc.dfu 
     39File: t1enc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 
    4040   defining Unicode char U+00A0 (decimal 160) 
    4141   defining Unicode char U+00A1 (decimal 161) 
     
    249249   defining Unicode char U+021A (decimal 538) 
    250250   defining Unicode char U+021B (decimal 539) 
    251    defining Unicode char U+1E02 (decimal 7682) 
    252    defining Unicode char U+1E03 (decimal 7683) 
     251   defining Unicode char U+01E02 (decimal 7682) 
     252   defining Unicode char U+01E03 (decimal 7683) 
    253253   defining Unicode char U+200C (decimal 8204) 
    254    defining Unicode char U+2010 (decimal 8208) 
    255    defining Unicode char U+2011 (decimal 8209) 
    256    defining Unicode char U+2012 (decimal 8210) 
    257254   defining Unicode char U+2013 (decimal 8211) 
    258255   defining Unicode char U+2014 (decimal 8212) 
    259    defining Unicode char U+2015 (decimal 8213) 
    260256   defining Unicode char U+2018 (decimal 8216) 
    261257   defining Unicode char U+2019 (decimal 8217) 
     
    273269... processing UTF-8 mapping file for font encoding OT1 
    274270 
    275 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/ot1enc.dfu 
    276 File: ot1enc.dfu 2016/02/28 v1.1s UTF-8 support for inputenc 
     271(/usr/share/texlive/texmf-dist/tex/latex/base/ot1enc.dfu 
     272File: ot1enc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 
    277273   defining Unicode char U+00A0 (decimal 160) 
    278274   defining Unicode char U+00A1 (decimal 161) 
     
    314310... processing UTF-8 mapping file for font encoding OMS 
    315311 
    316 (/usr/local/texlive/2016/texmf-dist/tex/latex/base/omsenc.dfu 
    317 File: omsenc.dfu 2016/02/28 v1.1s UTF-8 support for inputenc 
     312(/usr/share/texlive/texmf-dist/tex/latex/base/omsenc.dfu 
     313File: omsenc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 
    318314   defining Unicode char U+00A7 (decimal 167) 
    319315   defining Unicode char U+00B6 (decimal 182) 
     
    338334   defining Unicode char U+2423 (decimal 9251) 
    339335)) 
    340 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics/graphicx.sty 
     336(/usr/share/texlive/texmf-dist/tex/latex/graphics/graphicx.sty 
    341337Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR) 
    342338 
    343 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics/keyval.sty 
     339(/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty 
    344340Package: keyval 2014/10/28 v1.15 key=value parser (DPC) 
    345341\KV@toks@=\toks16 
    346342) 
    347 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics/graphics.sty 
    348 Package: graphics 2016/05/09 v1.0r Standard LaTeX Graphics (DPC,SPQR) 
    349  
    350 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics/trig.sty 
     343(/usr/share/texlive/texmf-dist/tex/latex/graphics/graphics.sty 
     344Package: graphics 2016/01/03 v1.0q Standard LaTeX Graphics (DPC,SPQR) 
     345 
     346(/usr/share/texlive/texmf-dist/tex/latex/graphics/trig.sty 
    351347Package: trig 2016/01/03 v1.10 sin cos tan (DPC) 
    352348) 
    353 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics-cfg/graphics.cfg 
    354 File: graphics.cfg 2016/01/02 v1.10 sample graphics configuration 
    355 ) 
    356 Package graphics Info: Driver file: pdftex.def on input line 96. 
    357  
    358 (/usr/local/texlive/2016/texmf-dist/tex/latex/pdftex-def/pdftex.def 
     349(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/graphics.cfg 
     350File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live 
     351) 
     352Package graphics Info: Driver file: pdftex.def on input line 95. 
     353 
     354(/usr/share/texlive/texmf-dist/tex/latex/pdftex-def/pdftex.def 
    359355File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX 
    360356 
    361 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/infwarerr.sty 
    362 Package: infwarerr 2016/05/16 v1.4 Providing info/warning/error messages (HO) 
    363 ) 
    364 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/ltxcmds.sty 
    365 Package: ltxcmds 2016/05/16 v1.23 LaTeX kernel commands for general use (HO) 
     357(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/infwarerr.sty 
     358Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO) 
     359) 
     360(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ltxcmds.sty 
     361Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO) 
    366362) 
    367363\Gread@gobject=\count87 
     
    370366\Gin@req@width=\dimen104 
    371367) 
    372 (/usr/local/texlive/2016/texmf-dist/tex/latex/listings/listings.sty 
     368(/usr/share/texlive/texmf-dist/tex/latex/listings/listings.sty 
    373369\lst@mode=\count88 
    374370\lst@gtempboxa=\box26 
     
    384380\lst@maxwidth=\dimen108 
    385381 
    386 (/usr/local/texlive/2016/texmf-dist/tex/latex/listings/lstmisc.sty 
     382(/usr/share/texlive/texmf-dist/tex/latex/listings/lstmisc.sty 
    387383File: lstmisc.sty 2015/06/04 1.6 (Carsten Heinz) 
    388384\c@lstnumber=\count94 
     
    390386\lst@framebox=\box27 
    391387) 
    392 (/usr/local/texlive/2016/texmf-dist/tex/latex/listings/listings.cfg 
     388(/usr/share/texlive/texmf-dist/tex/latex/listings/listings.cfg 
    393389File: listings.cfg 2015/06/04 1.6 listings configuration 
    394390)) 
    395391Package: listings 2015/06/04 1.6 (Carsten Heinz) 
    396392 
    397 (/usr/local/texlive/2016/texmf-dist/tex/latex/xcolor/xcolor.sty 
    398 Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK) 
    399  
    400 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics-cfg/color.cfg 
    401 File: color.cfg 2016/01/02 v1.6 sample color configuration 
     393(/usr/share/texlive/texmf-dist/tex/latex/xcolor/xcolor.sty 
     394Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK) 
     395 
     396(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/color.cfg 
     397File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive 
    402398) 
    403399Package xcolor Info: Package option `usenames' ignored on input line 216. 
    404400Package xcolor Info: Driver file: pdftex.def on input line 225. 
    405401 
    406 (/usr/local/texlive/2016/texmf-dist/tex/latex/colortbl/colortbl.sty 
     402(/usr/share/texlive/texmf-dist/tex/latex/colortbl/colortbl.sty 
    407403Package: colortbl 2012/02/13 v1.0a Color table columns (DPC) 
    408404 
    409 (/usr/local/texlive/2016/texmf-dist/tex/latex/tools/array.sty 
     405(/usr/share/texlive/texmf-dist/tex/latex/tools/array.sty 
    410406Package: array 2014/10/28 v2.4c Tabular extension package (FMi) 
    411407\col@sep=\dimen109 
     
    419415) 
    420416\rownum=\count96 
    421 Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348. 
    422 Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352. 
    423 Package xcolor Info: Model `RGB' extended on input line 1364. 
    424 Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366. 
    425 Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367. 
    426 Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368. 
    427 Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369. 
    428 Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370. 
    429 Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371. 
    430  
    431 (/usr/local/texlive/2016/texmf-dist/tex/latex/graphics/dvipsnam.def 
     417Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1337. 
     418Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1341. 
     419Package xcolor Info: Model `RGB' extended on input line 1353. 
     420Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1355. 
     421Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1356. 
     422Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1357. 
     423Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1358. 
     424Package xcolor Info: Model `Gray' substituted by `gray' on input line 1359. 
     425Package xcolor Info: Model `wave' substituted by `hsb' on input line 1360. 
     426 
     427(/usr/share/texlive/texmf-dist/tex/latex/graphics/dvipsnam.def 
    432428File: dvipsnam.def 2015/12/30 v3.0k Driver-dependent file (DPC,SPQR) 
    433429) 
    434 (/usr/local/texlive/2016/texmf-dist/tex/latex/xcolor/svgnam.def 
    435 File: svgnam.def 2016/05/11 v2.12 Predefined colors according to SVG 1.1 (UK) 
     430(/usr/share/texlive/texmf-dist/tex/latex/xcolor/svgnam.def 
     431File: svgnam.def 2007/01/21 v2.11 Predefined colors according to SVG 1.1 (UK) 
    436432)) 
    437 (/usr/local/texlive/2016/texmf-dist/tex/latex/amsmath/amsmath.sty 
    438 Package: amsmath 2016/03/10 v2.15b AMS math features 
     433(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty 
     434Package: amsmath 2016/03/03 v2.15a AMS math features 
    439435\@mathmargin=\skip46 
    440436 
    441437For additional information on amsmath, use the `?' option. 
    442 (/usr/local/texlive/2016/texmf-dist/tex/latex/amsmath/amstext.sty 
     438(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty 
    443439Package: amstext 2000/06/29 v2.01 AMS text 
    444440 
    445 (/usr/local/texlive/2016/texmf-dist/tex/latex/amsmath/amsgen.sty 
     441(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty 
    446442File: amsgen.sty 1999/11/30 v2.0 generic functions 
    447443\@emptytoks=\toks20 
    448444\ex@=\dimen111 
    449445)) 
    450 (/usr/local/texlive/2016/texmf-dist/tex/latex/amsmath/amsbsy.sty 
     446(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty 
    451447Package: amsbsy 1999/11/29 v1.2d Bold Symbols 
    452448\pmbraise@=\dimen112 
    453449) 
    454 (/usr/local/texlive/2016/texmf-dist/tex/latex/amsmath/amsopn.sty 
    455 Package: amsopn 2016/03/08 v2.02 operator names 
     450(/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty 
     451Package: amsopn 1999/12/14 v2.01 operator names 
    456452) 
    457453\inf@bad=\count97 
     
    468464\strutbox@=\box29 
    469465\big@size=\dimen113 
    470 LaTeX Font Info:    Redeclaring font encoding OML on input line 634. 
    471 LaTeX Font Info:    Redeclaring font encoding OMS on input line 635. 
     466LaTeX Font Info:    Redeclaring font encoding OML on input line 630. 
     467LaTeX Font Info:    Redeclaring font encoding OMS on input line 631. 
    472468\macc@depth=\count102 
    473469\c@MaxMatrixCols=\count103 
     
    490486\multlinetaggap=\skip48 
    491487\mathdisplay@stack=\toks24 
    492 LaTeX Info: Redefining \[ on input line 2739. 
    493 LaTeX Info: Redefining \] on input line 2740. 
    494 ) 
    495 (/usr/local/texlive/2016/texmf-dist/tex/latex/url/url.sty 
     488LaTeX Info: Redefining \[ on input line 2735. 
     489LaTeX Info: Redefining \] on input line 2736. 
     490) 
     491(/usr/share/texlive/texmf-dist/tex/latex/url/url.sty 
    496492\Urlmuskip=\muskip11 
    497493Package: url 2013/09/16  ver 3.4  Verb mode for urls, etc. 
    498494) 
     495(/usr/share/texlive/texmf-dist/tex/latex/tools/verbatim.sty 
     496Package: verbatim 2014/10/28 v1.5q LaTeX2e package for verbatim enhancements 
     497\every@verbatim=\toks25 
     498\verbatim@line=\toks26 
     499\verbatim@in@stream=\read1 
     500) 
    499501(./rapport ESIWACE.aux) 
    500502\openout1 = `"rapport ESIWACE.aux"'. 
    501503 
    502 LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 16. 
    503 LaTeX Font Info:    ... okay on input line 16. 
    504 LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 16. 
    505 LaTeX Font Info:    ... okay on input line 16. 
    506 LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 16. 
    507 LaTeX Font Info:    ... okay on input line 16. 
    508 LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 16. 
    509 LaTeX Font Info:    ... okay on input line 16. 
    510 LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 16. 
    511 LaTeX Font Info:    ... okay on input line 16. 
    512 LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 16. 
    513 LaTeX Font Info:    ... okay on input line 16. 
    514  
    515 (/usr/local/texlive/2016/texmf-dist/tex/context/base/mkii/supp-pdf.mkii 
     504LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 17. 
     505LaTeX Font Info:    ... okay on input line 17. 
     506LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 17. 
     507LaTeX Font Info:    ... okay on input line 17. 
     508LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 17. 
     509LaTeX Font Info:    ... okay on input line 17. 
     510LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 17. 
     511LaTeX Font Info:    ... okay on input line 17. 
     512LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 17. 
     513LaTeX Font Info:    ... okay on input line 17. 
     514LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 17. 
     515LaTeX Font Info:    ... okay on input line 17. 
     516 
     517(/usr/share/texlive/texmf-dist/tex/context/base/supp-pdf.mkii 
    516518[Loading MPS to PDF converter (version 2006.09.02).] 
    517519\scratchcounter=\count109 
     
    520522\nofMPsegments=\count110 
    521523\nofMParguments=\count111 
    522 \everyMPshowfont=\toks25 
     524\everyMPshowfont=\toks27 
    523525\MPscratchCnt=\count112 
    524526\MPscratchDim=\dimen121 
    525527\MPnumerator=\count113 
    526528\makeMPintoPDFobject=\count114 
    527 \everyMPtoPDFconversion=\toks26 
    528 ) (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty 
    529 Package: pdftexcmds 2016/05/10 v0.21 Utility functions of pdfTeX for LuaTeX (HO 
    530 ) 
    531  
    532 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/ifluatex.sty 
    533 Package: ifluatex 2016/05/16 v1.4 Provides the ifluatex switch (HO) 
     529\everyMPtoPDFconversion=\toks28 
     530) (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty 
     531Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO 
     532) 
     533 
     534(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifluatex.sty 
     535Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO) 
    534536Package ifluatex Info: LuaTeX not detected. 
    535537) 
    536 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/ifpdf.sty 
    537 Package: ifpdf 2016/05/14 v3.1 Provides the ifpdf switch 
     538(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifpdf.sty 
     539Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO) 
     540Package ifpdf Info: pdfTeX in PDF mode is detected. 
    538541) 
    539542Package pdftexcmds Info: LuaTeX not detected. 
     
    542545Package pdftexcmds Info: \pdfdraftmode found. 
    543546) 
    544 (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty 
    545 Package: epstopdf-base 2016/05/15 v2.6 Base part for package epstopdf 
    546  
    547 (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/grfext.sty 
    548 Package: grfext 2016/05/16 v1.2 Manage graphics extensions (HO) 
    549  
    550 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty 
    551 Package: kvdefinekeys 2016/05/16 v1.4 Define keys (HO) 
     547(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty 
     548Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf 
     549 
     550(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/grfext.sty 
     551Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO) 
     552 
     553(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty 
     554Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO) 
    552555)) 
    553 (/usr/local/texlive/2016/texmf-dist/tex/latex/oberdiek/kvoptions.sty 
    554 Package: kvoptions 2016/05/16 v3.12 Key value format for package options (HO) 
    555  
    556 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty 
    557 Package: kvsetkeys 2016/05/16 v1.17 Key value parser (HO) 
    558  
    559 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/etexcmds.sty 
    560 Package: etexcmds 2016/05/16 v1.6 Avoid name clashes with e-TeX commands (HO) 
     556(/usr/share/texlive/texmf-dist/tex/latex/oberdiek/kvoptions.sty 
     557Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO) 
     558 
     559(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty 
     560Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO) 
     561 
     562(/usr/share/texlive/texmf-dist/tex/generic/oberdiek/etexcmds.sty 
     563Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO) 
    561564Package etexcmds Info: Could not find \expanded. 
    562565(etexcmds)             That can mean that you are not using pdfTeX 1.50 or 
     
    567570(grfext)             [.png,.pdf,.jpg,.mps,.jpeg,.jbig2,.jb2,.PNG,.PDF,.JPG,.JPE 
    568571G,.JBIG2,.JB2,.eps] 
    569 (grfext)             \AppendGraphicsExtensions on input line 456. 
    570  
    571 (/usr/local/texlive/2016/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg 
     572(grfext)             \AppendGraphicsExtensions on input line 452. 
     573 
     574(/usr/share/texlive/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg 
    572575File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv 
    573576e 
     
    578581File: Charge1.png Graphic file (type png) 
    579582 <use Charge1.png> 
    580 Package pdftex.def Info: Charge1.png used on input line 32. 
     583Package pdftex.def Info: Charge1.png used on input line 33. 
    581584(pdftex.def)             Requested size: 165.01357pt x 91.23924pt. 
    582585 
     
    584587File: Charge2.png Graphic file (type png) 
    585588 <use Charge2.png> 
    586 Package pdftex.def Info: Charge2.png used on input line 33. 
     589Package pdftex.def Info: Charge2.png used on input line 34. 
    587590(pdftex.def)             Requested size: 165.25446pt x 91.05858pt. 
    588591 [1 
    589592 
    590 {/usr/local/texlive/2016/texmf-var/fonts/map/pdftex/updmap/pdftex.map} <./Charg 
    591 e1.png> <./Charge2.png>] <domain.pdf, id=16, 472.31454pt x 142.26149pt> 
     593{/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map} <./Charge1.png> <./Charge2. 
     594png>] 
     595<domain.pdf, id=16, 472.31454pt x 142.26149pt> 
    592596File: domain.pdf Graphic file (type pdf) 
    593597 <use domain.pdf> 
    594 Package pdftex.def Info: domain.pdf used on input line 59. 
     598Package pdftex.def Info: domain.pdf used on input line 60. 
    595599(pdftex.def)             Requested size: 236.1567pt x 71.13055pt. 
    596600 
     
    598602File: omp.pdf Graphic file (type pdf) 
    599603 <use omp.pdf> 
    600 Package pdftex.def Info: omp.pdf used on input line 67. 
     604Package pdftex.def Info: omp.pdf used on input line 68. 
    601605(pdftex.def)             Requested size: 291.64784pt x 126.32893pt. 
    602  
    603  
    604 LaTeX Warning: `!h' float specifier changed to `!ht'. 
    605  
    606 [2 <./domain.pdf>] <scheme.png, id=27, 665.48625pt x 517.935pt> 
     606 [2 <./domain.pdf>] 
     607<scheme.png, id=27, 665.48625pt x 517.935pt> 
    607608File: scheme.png Graphic file (type png) 
    608  
    609 <use scheme.png> 
    610 Package pdftex.def Info: scheme.png used on input line 79. 
     609 <use scheme.png> 
     610Package pdftex.def Info: scheme.png used on input line 86. 
    611611(pdftex.def)             Requested size: 266.18977pt x 207.17032pt. 
    612  [3 <./omp.pdf> <./scheme.png (PNG copy)>] [4] 
    613 <LMDZ_perf.png, id=39, 530.73282pt x 255.95625pt> 
    614 File: LMDZ_perf.png Graphic file (type png) 
    615  <use LMDZ_perf.png> 
    616 Package pdftex.def Info: LMDZ_perf.png used on input line 144. 
    617 (pdftex.def)             Requested size: 318.44214pt x 153.57492pt. 
    618  [5 <./LMDZ_perf.png (PNG copy)>] (./rapport ESIWACE.bbl) [6] (./rapport ESIWAC 
    619 E.aux) )  
     612 [3 <./omp.pdf>] 
     613<tag.png, id=36, 752.8125pt x 103.38625pt> 
     614File: tag.png Graphic file (type png) 
     615 <use tag.png> 
     616Package pdftex.def Info: tag.png used on input line 140. 
     617(pdftex.def)             Requested size: 301.11966pt x 41.35376pt. 
     618 [4 <./scheme.png (PNG copy)>] <sendrecv.png, id=41, 829.0975pt x 694.595pt> 
     619File: sendrecv.png Graphic file (type png) 
     620 <use sendrecv.png> 
     621Package pdftex.def Info: sendrecv.png used on input line 149. 
     622(pdftex.def)             Requested size: 331.63313pt x 277.83307pt. 
     623 
     624[5 <./tag.png (PNG copy)> <./sendrecv.png (PNG copy)>] 
     625<bcast.png, id=45, 512.91624pt x 552.0625pt> 
     626File: bcast.png Graphic file (type png) 
     627 <use bcast.png> 
     628Package pdftex.def Info: bcast.png used on input line 173. 
     629(pdftex.def)             Requested size: 153.87605pt x 165.62003pt. 
     630 
     631<allreduce.png, id=46, 743.77875pt x 552.0625pt> 
     632File: allreduce.png Graphic file (type png) 
     633 <use allreduce.png> 
     634Package pdftex.def Info: allreduce.png used on input line 185. 
     635(pdftex.def)             Requested size: 223.13535pt x 165.62003pt. 
     636 [6 <./bcast.png (PNG copy)> <./allreduce.png (PNG copy)>] (./rapport ESIWACE.b 
     637bl 
     638Underfull \hbox (badness 3354) in paragraph at lines 4--9 
     639[]\OT1/cmr/m/n/10 XIOS de-vel-op-per group.  Note for XIOS End-points.  Tech-ni 
     640-cal 
     641 [] 
     642 
     643 
     644Underfull \hbox (badness 4819) in paragraph at lines 4--9 
     645\OT1/cmr/m/n/10 re-port, $\OT1/cmtt/m/n/10 http : / / forge . ipsl . jussieu .  
     646fr / ioserver / browser / XIOS / dev / 
     647 [] 
     648 
     649) [7] (./rapport ESIWACE.aux) )  
    620650Here is how much of TeX's memory you used: 
    621  4595 strings out of 493014 
    622  61054 string characters out of 6133351 
    623  142700 words of memory out of 5000000 
    624  8068 multiletter control sequences out of 15000+600000 
     651 4637 strings out of 494953 
     652 61251 string characters out of 6180977 
     653 137489 words of memory out of 5000000 
     654 7849 multiletter control sequences out of 15000+600000 
    625655 9090 words of font info for 34 fonts, out of 8000000 for 9000 
    626  1141 hyphenation exceptions out of 8191 
     656 14 hyphenation exceptions out of 8191 
    627657 41i,8n,35p,1270b,264s stack positions out of 5000i,500n,10000p,200000b,80000s 
    628 </usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmbx 
    629 12.pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi1 
    630 0.pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi7. 
    631 pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pf 
    632 b></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr12.pfb> 
    633 </usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr17.pfb></ 
    634 usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr6.pfb></usr 
    635 /local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr7.pfb></usr/lo 
    636 cal/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr8.pfb></usr/local 
    637 /texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb></usr/local/ 
    638 texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmti10.pfb></usr/local/t 
    639 exlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/local/te 
    640 xlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt8.pfb> 
    641 Output written on "rapport ESIWACE.pdf" (6 pages, 229957 bytes). 
     658</usr/share/texlive/texmf-dist/fonts/type1/publi 
     659c/amsfonts/cm/cmbx12.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsf 
     660onts/cm/cmr10.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm 
     661/cmr12.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr17. 
     662pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr6.pfb></us 
     663r/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr7.pfb></usr/share/ 
     664texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr8.pfb></usr/share/texlive/ 
     665texmf-dist/fonts/type1/public/amsfonts/cm/cmti10.pfb></usr/share/texlive/texmf- 
     666dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texlive/texmf-dist/f 
     667onts/type1/public/amsfonts/cm/cmtt8.pfb> 
     668Output written on "rapport ESIWACE.pdf" (7 pages, 269577 bytes). 
    642669PDF statistics: 
    643  90 PDF objects out of 1000 (max. 8388607) 
    644  62 compressed objects within 1 object stream 
     670 86 PDF objects out of 1000 (max. 8388607) 
     671 57 compressed objects within 1 object stream 
    645672 0 named destinations out of 1000 (max. 500000) 
    646  31 words of extra memory for PDF output out of 10000 (max. 10000000) 
    647  
     673 46 words of extra memory for PDF output out of 10000 (max. 10000000) 
     674 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.tex

    r1551 r1552  
    66\usepackage{amsmath} 
    77\usepackage{url} 
     8\usepackage{verbatim} 
    89 
    910% Title Page 
     
    2930writing on the server side can be overlapped completely by calculates on the client side.  
    3031 
    31 \begin{figure}[h] 
     32\begin{figure}[ht] 
    3233\includegraphics[scale=0.4]{Charge1.png} 
    3334\includegraphics[scale=0.4]{Charge2.png} 
     
    5556This topic comes along with the configuration of the climate models. Take LMDZ as example, it is designed with the 2-level parallelization scheme. To be more specific, LMDZ uses the domain decomposition method in which each sub-domain is associated with one MPI process. Inside of the sub-domain, the model also uses OpenMP derivatives to accelerate the computation. We can imagine that the sub-domain be divided into sub-sub-domain and is managed by threads.  
    5657 
    57 \begin{figure}[h] 
     58\begin{figure}[ht] 
    5859\centering 
    5960\includegraphics[scale=0.5]{domain.pdf} 
     
    6364As we know, each sub-domain, or in another word, each MPI process is a XIOS client. The data exchange between client and XIOS servers is handled by MPI communications. In order to write an output field, all threads must gather the data to the master thread who acts as MPI process in order to call MPI routines. There are two disadvantages about this method : first, we have to spend time on gathering information to the master thread which not only increases the memory use, but also implies an OpenMP barrier; second, while the master thread calls MPI routine, other threads are in the idle state thus a waster of computing resources. What we want obtain with the thread-friendly XIOS is that all threads can act like MPI processes. They can call directly the MPI routine thus no waste in memory nor in computing resources as shown in Figure \ref{fig:omp}. 
    6465 
    65 \begin{figure}[h!] 
     66\begin{figure}[ht] 
    6667\centering 
    6768\includegraphics[scale=0.6]{omp.pdf} 
     
    7374 
    7475 
    75 In this project, we choose to implement the interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a concept proposed in the last MPI Forum and several papers has already discussed the importance of such idea and have introduced the framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. Threads of an MPI process is associated with a unique rank (global endpoint rank) and an endpoint communicator. They also have a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num| rank.  
    76  
    77 \begin{figure}[h!] 
     76In this project, we choose to implement an interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a  
     77concept proposed in the last MPI Forums and several papers have already discussed the importance of such idea and have introduced the  
     78framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. In  
     79the MPI\_endpoint environment, each OpenMP thread will be associated with a unique rank (global endpoint rank), an endpoint communicator,  
     80and a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num|. The global endpoint rank will replace the  
     81role of the classic MPI rank and will be used in MPI communication calls.  
     82 
     83 
     84\begin{figure}[ht] 
    7885\begin{center} 
    7986\includegraphics[scale=0.4]{scheme.png}  
     
    8592%XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we introduce the MPI endpoints.   
    8693 
    87  
    88 The MPI\_endpoints interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions used in XIOS.  
    89  
    90 will be re-implemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be  
    91 associated with a unique rank and with an endpoint communicator. This rank (EP rank) will replace the role of the classic MPI rank and will  
    92 be used in MPI communications. In order to successfully execute an MPI communication, for example \verb|MPI_Send|, we know already which  
    93 endpoints to be the receiver but not sufficient. We also need to know which MPI process should be involved in such communication. To  
    94 identify the MPI rank, we added a ``map'' in the EP communicator in which the relation of all EP and MPI ranks can be easily obtained. 
    95  
    96  
    97 In XIOS, we used the ``probe'' technique to search for arrived messages and then performing the receive action. The principle is  
    98 that sender processes execute the send operations as usual. However, to minimize the time spent on waiting incoming messages, the receiver  
    99 processe performs in the first place the \verb|MPI_Probe| function to check if a message destinated to it has been published. If yes, the  
    100 process execute in the second place the \verb|MPI_Recv| to receive the message. In this situation, if we introduce the threads, problems  
    101 occur. The reason why the ``probe'' method is not suitable is that messages destinated to one certain process can be probed by any of  
    102 its threads. Thus the message can be received by the wrong thread which gives errors. 
    103  
    104 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each process is equiped with a local  
    105 incoming message queue. All incoming message will be probed, sorted, and then stored in this queue according to their destination rank.  
    106 Every time we call an MPI function, we firstly call the \verb|MPI_Mprobe| function to get the handle to  
    107 the incoming message. Then, we identify the destination thread rank and store the message handle inside the local queue of the target  
    108 thread. After this, we perform the usual ``probe'' technique upon the local incoming message queue. In this way, we can assure the messages  
    109 to be received by the right thread. 
    110  
    111 Another issue remains in this technique: how to identify the receiver's rank? The solution is to use the tag argument. In the MPI  
    112 environment, a tag is an integer ranging from 0 to $2^{31}$. We can explore the large range of the tag to store in it information about the  
    113 source and destination thread ranks. We choose to limite the first 15 bits for the tag used in the classic MPI communication, the next 8  
    114 bits to the sender's thread rank, and the last 8 bits to the receiver's thread rank. In such way, with an extra analysis of the EP tag, we  
    115 can identify the ranks of the sender and the receiver in any P2P communication. As results, we a thread probes a message, it knows  
    116 exactly in which local queue should store the probed message.  
    117  
    118  
    119 With the global rank map, tag extension, and the matching-probe techniques, we are able to use any P2P communication in the endpoint  
    120 environment. For the collective communications, we perform a step-by-step execution and no special technique is required. The most  
    121 representative functions is the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. A step-by-step execution consists of  
    122 3 steps (not necessarily in this order): arrangement of the source data, execution of the MPI function by all  
    123 master/root threads, distribution or arrangement of the data among threads.  
    124  
    125 For example, if we want to perform a broadcast operation, 2 steps are needed. Firstly, the root thread, along with the master threads of  
    126 other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the master threads send data to threads  
    127 sharing the same process via local memory transfer. In another example for illustrating the \verb|MPI_Gather| function, we also need 2  
    128 steps. First of all, data is gathered from slave threads to the master thread or the root thread. Next, the master thread and the root  
    129 thread execute the \verb|MPI_Gather| operation of complete the communication. Other collective calls such as \verb|MPI_Scan|,  
    130 \verb|MPI_Reduce|, \verb|MPI_Scatter| \textit{etc} follow the same principle of step-by-step execution. 
    131  
    132  
     94An other important aspect about the MPI\_endpoint interface is that each endpoints has knowledge of the ranks of other endpoints in the  
     95same communicator. This knowledge is necessary because when executing an MPI communication, for example a point-to-point exchange, we need  
     96to know not only the ranks of sender/receiver threads, but also the thread number of the sender/receiver threads and the MPI ranks  
     97of the sender/receiver processes. This ranking information is implemented inside an map object included in the endpoint communicator  
     98class. 
     99 
     100 
     101 
     102 
     103%\newpage 
     104 
     105%The MPI\_endpoint interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions  
     106%used in XIOS.  
     107 
     108 
     109In XIOS, we used the ``probe'' technique to search for arrived messages and then perform the receiving action. The principle is  
     110that sender process executes the send operation as usual. However, to minimize the time spent on waiting incoming messages, the receiver  
     111process calls in the first place the \verb|MPI_Probe| function to check if a message destinate to it has been published. If yes, the  
     112process execute in the second place the \verb|MPI_Recv| to receive the message. If not, the receiver process can carry on with other tasks  
     113or repeats the \verb|MPI_Probe| and \verb|MPI_Recv| actions if the required message is in immediate need. This technique works well in  
     114the current version of XIOS. However, if we introduce threads into this mechanism, problems can occur: The incoming message is labeled by  
     115the tag and receiver's MPI rank. Because threads within a process share the MPI rank, and the message probed is always available in the  
     116message queue, it can lead to the problem of data race and thus the message can be received by the wrong thread. 
     117 
     118 
     119To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each thread is equipped with a local  
     120incoming message queue. Each time a thread calls an MPI function, for example \verb|MPI_Recv|, it calls firstly the \verb|MPI_Mprobe|  
     121function to query the MPI incoming message with any tag and from any source. Once a message is probed, the thread gets the handle to the  
     122incoming message and this specific message is erased from the MPI message queue. Then, the thread proceed the identification of the message  
     123to get the destination thread's local rank and store the message handle to the local queue of the target thread. The thread repeats these  
     124steps until the MPI incoming message queue is empty. Then the thread we perform the usual ``probe'' technique to query its local incoming  
     125message queue to check if the required message is available. If yes, it performs the \verb|MPI_Recv| operation. With this ``matching-probe''  
     126technique, we can assure that a message is probed only once and is received by the right receiver. 
     127  
     128 
     129 
     130Another issue needs to be clarified with this technique is that: how to identify the receiver's rank? The solution to this question is to  
     131use the tag argument. In the MPI environment, a tag is an integer ranging from 0 to $2^{31}$ depending on the Implementation. We can explore  
     132the large range property of the tag to store in it information about the source and destination thread ranks. In our endpoint interface, we  
     133choose to limit the first 15 bits for the tag used in the classic MPI communication, the next 8 bits to store the sender thread's local  
     134rank, and the last 8 bits to store the receiver thread's local rank (\textit{c.f.} Figure \ref{fig:tag}). In this way, with an extra  
     135analysis of the tag, we can identify the local ranks of the sender and the receiver in any P2P communication. As results, when a thread  
     136probes a message, it knows exactly in which local queue should store the probed message.  
     137 
     138\begin{figure}[ht] 
     139 \centering 
     140 \includegraphics[scale=0.4]{tag.png} 
     141 \caption{}\label{fig:tag} 
     142\end{figure} 
     143 
     144In Figure \ref{fig:tag}, Tag contains the user defined value for a certain communication. MPI\_tag is computed in the endpoint interface  
     145with help of the rank map and is used in the MPI calls.  
     146 
     147\begin{figure}[ht] 
     148\centering 
     149 \includegraphics[scale = 0.4]{sendrecv.png} 
     150\caption{This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message  
     151to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1  
     152and tag=65537. From the receiver's point of view, the endpoint 3 is actually receiving a message from MPI rank 0 with  
     153tag=65537.} 
     154\label{fig:sendrecv} 
     155\end{figure} 
     156 
     157 
     158 
     159 
     160With the rank map, tag extension, and the matching-probe techniques, we are now able to call any P2P communication in the endpoint  
     161environment. For the collective communications, we apply a step-by-step execution pattern and no special technique is required. A  
     162step-by-step execution pattern consists of 3 steps (not necessarily in this order and not all steps are needed): arrangement of the source  
     163data, execution of the MPI function by all master/root threads, distribution or arrangement of the resulting data among threads.  
     164 
     165%The most representative functions of the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. 
     166 
     167For example, if we want to perform a broadcast operation, only 2 steps are needed (\textit{c.f.} Figure \ref{fig:bcast}). Firstly, the root  
     168thread, along with the master threads of other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the  
     169master threads send data to other threads via local memory transfer.  
     170 
     171\begin{figure}[ht] 
     172\centering 
     173\includegraphics[scale=0.3]{bcast.png}  
     174\caption{} 
     175\label{fig:bcast} 
     176\end{figure} 
     177 
     178Figure \ref{fig:allreduce} illustrates how the \verb|MPI_Allreduce| function is proceeded in the endpoint interface. First of all, We  
     179perform a intra-process ``allreduce'' operation: source data is reduced from slave threads to the master thread via local memory transfer.  
     180Next, all master threads call the classic \verb|MPI_Allreduce| routine. Finally, all master threads send the updated reduced data to its  
     181slaves via local memory transfer.  
     182 
     183\begin{figure}[ht] 
     184\centering 
     185\includegraphics[scale=0.3]{allreduce.png}  
     186\caption{} 
     187\label{fig:allreduce} 
     188\end{figure} 
     189 
     190Other MPI routines, such as \verb|MPI_Wait|, \verb|MPI_Intercomm_create| \textit{etc.}, can be found in the technique report of the  
     191endpoint interface. 
     192 
     193\section{The multi-threaded XIOS and performance results} 
     194 
     195The development of endpoint interface for thread-friendly XIOS library took about one year and a half. The main difficulty is the  
     196co-existence of MPI processes and OpenMP threads. All MPI classes must be redefined in the endpoint interface along with all the routines.  
     197The development is now available on the forge server: \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp}. One  
     198technique report is also available in which one can find more detail about how endpoint works and how the routines are implemented  
     199\cite{ep:2018}. We must note that the thread-friendly XIOS library is still in the phase of optimization. It will be released in the  
     200future with a stable version. 
     201 
     202All the functionalities of XIOS is reserved in its thread-friendly version. Single threaded code can work successfully with the new  
     203version of XIOS. For multi-threaded models, some modifications are needed in order to work with the multi-threaded XIOS library. Detail can  
     204be found in our technique report \cite{ep:2018}. 
     205 
     206Even though the multi-threaded XIOS library is not fully accomplished and further optimization in ongoing. We have already done some tests  
     207to see the potential of the endpoint framework. We take LMDZ as the target model and have tested with several work-flow charges.  
     208 
     209\begin{comment} 
    133210\section{Performance of LMDZ using EP\_XIOS} 
    134211 
     
    141218simulation duration settings: 1 day, 5 days, 15 days, and 31 days.  
    142219 
    143 \begin{figure}[h] 
     220\begin{figure}[ht] 
    144221 \centering 
    145222 \includegraphics[scale = 0.6]{LMDZ_perf.png} 
     
    167244histmth with daily output 
    168245 
    169 \section{Perspectives of EP\_XIOS} 
     246\end{comment} 
     247 
     248 
     249\section{Future works for XIOS} 
    170250 
    171251 
  • XIOS/dev/branch_openmp/Note/rapport ESIWACE.tex.backup

    r1548 r1552  
    55\usepackage[usenames,dvipsnames,svgnames,table]{xcolor} 
    66\usepackage{amsmath} 
     7\usepackage{url} 
    78 
    89% Title Page 
    910 
    10 \title{Developping XIOS with multithread : to accelerate the IO of climate models} 
     11\title{Developing XIOS with multi-thread : to accelerate the I/O of climate models} 
    1112 
    1213\author{} 
     
    1617\maketitle 
    1718 
    18 \section{background} 
     19\section{Context} 
    1920 
    2021The simulation models of climate systems, running on a large number of computing resources can produce an important volume of data. At this  
    21 scale, the IO and the post-treatement of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux  
    22 generated by the simulations, we use XIOS developped by the Institut Pierre Simon Laplace and Maison de la simulation. 
    23  
    24 XIOS, a libarary dedicated to intense calculates, allows us to easily and efficiently manage the parallel IO on the storage systems. XIOS  
     22scale, the I/O and the post-treatment of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux  
     23generated by the simulations, we use XIOS developed by the Institut Pierre Simon Laplace and Maison de la simulation. 
     24 
     25XIOS, a library dedicated to intense calculates, allows us to easily and efficiently manage the parallel I/O on the storage systems. XIOS  
    2526uses the client/server scheme in which computing resources (server) are reserved exclusively for IO in order to minimize their impact on  
    26 the performance of the climate models (client). 
    27  
    28 Cette bibliothÚque, dédiée au calcul intensif, permet de gérer efficacement et simplement les 
    29 entrée/sortie parallÚles des données sur les systÚmes de stockage. Dans cette nouvelle 
    30 approche, orientée client/serveur, des cœurs de calcul sont exclusivement dédiés aux I/O de 
    31 façon à minimiser leur impact sur le temps de calcul des modÚles. L’utilisation des 
    32 communications asynchrones entre les modÚles (clients) et les serveurs I/O permet de lisser 
    33 les pics I/O en envoyant un flux de données constant au systÚme de fichiers tout au long de la 
    34 simulation, recouvrant ainsi totalement les écritures par du calcul. 
    35  
    36  
    37 The aim of this project ESIWACE is to develop a multithreaded version of XIOS, a library dedicated to IO manegement of climate code. 
    38 The current XIOS code lies on a single level of parallelization using MPI. However, many climate models are now disigned with two-level  
    39 parallelization through MPI and OpenMP. The difference of parallelization between the climate models and XIOS can lead to performance lost  
    40 because XIOS can not cope with threads. This fact  
    41  
    42  
    43 The resulting multithreaded XIOS is desinged to cope with climate models which use a two-level parallelization (MPI/Openmp) scheme. 
    44 The principle model we work with is the LMDZ code developped at Laboratoire de Météorologie Dynamique. This model has  
    45  
    46  
    47  
    48 \section{Developpement of a thread-friendly MPI for XIOS} 
    49  
    50 XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations  
    51 and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. 
    52 However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP  
    53 directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to  
    54 maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI  
    55 process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the  
    56 updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines.  
    57 This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a  
    58 new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we  
    59 introduce the MPI endpoints.   
    60  
    61  
    62 The MPI endpoints (EP) is a layer on top of an existing MPI Implementation. All MPI function, or in our work the functions used in XIOS,  
    63 will be reimplemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be  
    64 associated with a unique rank and with an endpoint communicator. This rank (EP rank) will replace the role of the classic MPI rank and will  
    65 be used in MPI communications. In order to successfully execute an MPI communication, for example \verb|MPI_Send|, we know already which  
    66 endpoints to be the receiver but not sufficient. We also need to know which MPI process should be involved in such communication. To  
    67 identify the MPI rank, we added a ``map'' in the EP communicator in which the relation of all EP and MPI ranks can be easily obtained. 
    68  
    69  
    70 In XIOS, we used the ``probe'' technique to search for arrived messages and then performing the receive action. The principle is  
    71 that sender processes execute the send operations as usual. However, to minimise the time spent on waiting incoming messages, the receiver  
    72 processe performs in the first place the \verb|MPI_Probe| function to check if a message destinated to it has been published. If yes, the  
    73 process execute in the second place the \verb|MPI_Recv| to receive the message. In this situation, if we introduce the threads, problems  
    74 occur. The reason why the ``probe'' method is not suitable is that messages destinated to one certain process can be probed by any of  
    75 its threads. Thus the message can be received by the wrong thread which gives errors. 
    76  
    77 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each process is equiped with a local  
    78 incoming message queue. All incoming message will be probed, sorted, and then stored in this queue according to their destination rank.  
    79 Every time we call an MPI function, we firstly call the \verb|MPI_Mprobe| function to get the handle to  
    80 the incoming message. Then, we identify the destination thread rank and store the message handle inside the local queue of the target  
    81 thread. After this, we perform the usual ``probe'' technique upon the local incoming message queue. In this way, we can assure the messages  
    82 to be received by the right thread. 
    83  
    84 Another issue remains in this technique: how to identify the receiver's rank? The solution is to use the tag argument. In the MPI  
    85 environment, a tag is an integer ranging from 0 to $2^{31}$. We can explore the large range of the tag to store in it information about the  
    86 source and destination thread ranks. We choose to limite the first 15 bits for the tag used in the classic MPI communication, the next 8  
    87 bits to the sender's thread rank, and the last 8 bits to the receiver's thread rank. In such way, with an extra analysis of the EP tag, we  
    88 can identify the ranks of the sender and the receiver in any P2P communication. As results, we a thread probes a message, it knows  
    89 exactly in which local queue should store the probed message.  
    90  
    91  
    92 With the global rank map, tag extension, and the matching-probe techniques, we are able to use any P2P communication in the endpoint  
    93 environment. For the collective communications, we perform a step-by-step execution and no special technique is required. The most  
    94 representative functions is the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. A step-by-step execution consists of  
    95 3 steps (not necessarily in this order): arrangement of the source data, execution of the MPI function by all  
    96 master/root threads, distribution or arrangement of the data among threads.  
    97  
    98 For example, if we want to perform a broadcast operation, 2 steps are needed. Firstly, the root thread, along with the master threads of  
    99 other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the master threads send data to threads  
    100 sharing the same process via local memory transfer. In another example for illustrating the \verb|MPI_Gather| function, we also need 2  
    101 steps. First of all, data is gathered from slave threads to the master thread or the root thread. Next, the master thread and the root  
    102 thread execute the \verb|MPI_Gather| operation of complete the communication. Other collective calls such as \verb|MPI_Scan|,  
    103 \verb|MPI_Reduce|, \verb|MPI_Scatter| \textit{etc} follow the same principle of step-by-step execution. 
    104  
     27the performance of the climate models (client). The clients and servers are executed in parallel and communicate asynchronously. In this  
     28way, the I/O peaks can be smoothed out as data fluxes are send to server constantly throughout the simulation and the time spent on data  
     29writing on the server side can be overlapped completely by calculates on the client side.  
     30 
     31\begin{figure}[ht] 
     32\includegraphics[scale=0.4]{Charge1.png} 
     33\includegraphics[scale=0.4]{Charge2.png} 
     34\caption{On the left, each peak of computing power corresponds to the valley of memory bandwidth which shows that the computing resources  
     35are alternating between calculates and I/O. ON the right, both curves are smooth which means that the computing resources have a stable  
     36charge of work, either calculates or I/O.} 
     37\end{figure} 
     38 
     39 
     40XIOS works well with many climate simulation codes. For example, LMDZ\footnote{LMDZ is a general circulation model (or global climate model)  
     41developed since the 70s at the "Laboratoire de Météorologie Dynamique", which includes various variants for the Earth and other planets  
     42(Mars, Titan, Venus, Exoplanets). The 'Z' in LMDZ stands for "zoom" (and the 'LMD' is for  'Laboratoire de Météorologie Dynamique").  
     43\url{http://lmdz.lmd.jussieu.fr}}, NENO\footnote{Nucleus for European Modeling of the Ocean alias NEMO is a  
     44state-of-the-art modelling framework of ocean related engines. \url{https://www.nemo-ocean.eu}}, ORCHIDEE\footnote{the land surface  
     45model of the IPSL (Institut Pierre Simon Laplace) Earth System Model. \url{https://orchidee.ipsl.fr}}, and DYNAMICO\footnote{The DYNAMICO  
     46project develops a new dynamical core for LMD-Z, the atmospheric general circulation model (GCM) part of IPSL-CM Earth System Model.  
     47\url{http://www.lmd.polytechnique.fr/~dubos/DYNAMICO/}} all use XIOS as the output back end. M\'et\'eoFrance and MetOffice also choose XIOS  
     48to manege the I/O for their models. 
     49 
     50 
     51\section{Development of thread-friendly XIOS} 
     52 
     53Although XIOS copes well with many models, there is one potential optimization in XIOS which needs to be investigated: making XIOS thread-friendly. 
     54 
     55This topic comes along with the configuration of the climate models. Take LMDZ as example, it is designed with the 2-level parallelization scheme. To be more specific, LMDZ uses the domain decomposition method in which each sub-domain is associated with one MPI process. Inside of the sub-domain, the model also uses OpenMP derivatives to accelerate the computation. We can imagine that the sub-domain be divided into sub-sub-domain and is managed by threads.  
     56 
     57\begin{figure}[ht] 
     58\centering 
     59\includegraphics[scale=0.5]{domain.pdf} 
     60\caption{Illustration of the domain decomposition used in LMDZ.} 
     61\end{figure} 
     62 
     63As we know, each sub-domain, or in another word, each MPI process is a XIOS client. The data exchange between client and XIOS servers is handled by MPI communications. In order to write an output field, all threads must gather the data to the master thread who acts as MPI process in order to call MPI routines. There are two disadvantages about this method : first, we have to spend time on gathering information to the master thread which not only increases the memory use, but also implies an OpenMP barrier; second, while the master thread calls MPI routine, other threads are in the idle state thus a waster of computing resources. What we want obtain with the thread-friendly XIOS is that all threads can act like MPI processes. They can call directly the MPI routine thus no waste in memory nor in computing resources as shown in Figure \ref{fig:omp}. 
     64 
     65\begin{figure}[ht] 
     66\centering 
     67\includegraphics[scale=0.6]{omp.pdf} 
     68\caption{} 
     69\label{fig:omp} 
     70\end{figure} 
     71 
     72There are two ways to make XIOS thread-friendly. First of all, change the structure of XIOS which demands a lot of modification is the XIOS library. Knowing that XIOS is about 100 000 lines of code, this method will be very time consuming. What's more, the modification will be local to XIOS. If we want to optimize an other code to be thread-friendly, we have to redo the modifications. The second choice is to add an extra interface to MPI in order to manage the threads. When a thread want to call an MPI routine inside XIOS, it will first pass the interface, in which the communication information will be analyzed before the MPI routine is invoked. With this method, we only need to modify a very small part of XIOS in order to make it work. What is more interesting is that the interface we created can be adjusted to suit other MPI based libraries. 
     73 
     74 
     75In this project, we choose to implement an interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a  
     76concept proposed in the last MPI Forums and several papers have already discussed the importance of such idea and have introduced the  
     77framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. In  
     78the MPI\_endpoint environment, each OpenMP thread will be associated with a unique rank (global endpoint rank), an endpoint communicator,  
     79and a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num|. The global endpoint rank will replace the  
     80role of the classic MPI rank and will be used in MPI communication calls.  
     81 
     82 
     83\begin{figure}[ht] 
     84\begin{center} 
     85\includegraphics[scale=0.4]{scheme.png}  
     86\end{center} 
     87\caption{} 
     88\label{fig:scheme} 
     89\end{figure} 
     90 
     91%XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we introduce the MPI endpoints.   
     92 
     93An other important aspect about the MPI\_endpoint interface is that each endpoints has knowledge of the ranks of other endpoints in the  
     94same communicator. This knowledge is necessary because when executing an MPI communication, for example a point-to-point exchange, we need  
     95to know not only the ranks of sender/receiver threads, but also the thread number of the sender/receiver threads and the MPI ranks  
     96of the sender/receiver processes. This ranking information is implemented inside an map object included in the endpoint communicator  
     97class. 
     98 
     99 
     100 
     101 
     102%\newpage 
     103 
     104%The MPI\_endpoint interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions  
     105%used in XIOS.  
     106 
     107 
     108In XIOS, we used the ``probe'' technique to search for arrived messages and then perform the receiving action. The principle is  
     109that sender process executes the send operation as usual. However, to minimize the time spent on waiting incoming messages, the receiver  
     110process calls in the first place the \verb|MPI_Probe| function to check if a message destinate to it has been published. If yes, the  
     111process execute in the second place the \verb|MPI_Recv| to receive the message. If not, the receiver process can carry on with other tasks  
     112or repeats the \verb|MPI_Probe| and \verb|MPI_Recv| actions if the required message is in immediate need. This technique works well in  
     113the current version of XIOS. However, if we introduce threads into this mechanism, problems can occur: The incoming message is labeled by  
     114the tag and receiver's MPI rank. Because threads within a process share the MPI rank, and the message probed is always available in the  
     115message queue, it can lead to the problem of data race and thus the message can be received by the wrong thread. 
     116 
     117 
     118To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each thread is equipped with a local  
     119incoming message queue. Each time a thread calls an MPI function, for example \verb|MPI_Recv|, it calls firstly the \verb|MPI_Mprobe|  
     120function to query the MPI incoming message with any tag and from any source. Once a message is probed, the thread gets the handle to the  
     121incoming message and this specific message is erased from the MPI message queue. Then, the thread proceed the identification of the message  
     122to get the destination thread's local rank and store the message handle to the local queue of the target thread. The thread repeats these  
     123steps until the MPI incoming message queue is empty. Then the thread we perform the usual ``probe'' technique to query its local incoming  
     124message queue to check if the required message is available. If yes, it performs the \verb|MPI_Recv| operation. With this ``matching-probe''  
     125technique, we can assure that a message is probed only once and is received by the right receiver. 
     126  
     127 
     128 
     129Another issue needs to be clarified with this technique is that: how to identify the receiver's rank? The solution to this question is to  
     130use the tag argument. In the MPI environment, a tag is an integer ranging from 0 to $2^{31}$ depending on the Implementation. We can explore  
     131the large range property of the tag to store in it information about the source and destination thread ranks. In our endpoint interface, we  
     132choose to limit the first 15 bits for the tag used in the classic MPI communication, the next 8 bits to store the sender thread's local  
     133rank, and the last 8 bits to store the receiver thread's local rank (\textit{c.f.} Figure \ref{fig:tag}). In this way, with an extra  
     134analysis of the tag, we can identify the local ranks of the sender and the receiver in any P2P communication. As results, when a thread  
     135probes a message, it knows exactly in which local queue should store the probed message.  
     136 
     137\begin{figure}[ht] 
     138 \centering 
     139 \includegraphics[scale=0.4]{tag.png} 
     140 \caption{}\label{fig:tag} 
     141\end{figure} 
     142 
     143In Figure \ref{fig:tag}, Tag contains the user defined value for a certain communication. MPI\_tag is computed in the endpoint interface  
     144with help of the rank map and is used in the MPI calls.  
     145 
     146\begin{figure}[ht] 
     147\centering 
     148 \includegraphics[scale = 0.4]{sendrecv.png} 
     149\caption{This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message  
     150to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1  
     151and tag=65537. From the receiver's point of view, the endpoint 3 is actually receving a message from MPI rank 0 with  
     152tag=65537.} 
     153\label{fig:sendrecv} 
     154\end{figure} 
     155 
     156 
     157 
     158 
     159With the rank map, tag extension, and the matching-probe techniques, we are now able to call any P2P communication in the endpoint  
     160environment. For the collective communications, we apply a step-by-step execution pattern and no special technique is required. A  
     161step-by-step execution pattern consists of 3 steps (not necessarily in this order and not all steps are needed): arrangement of the source  
     162data, execution of the MPI function by all master/root threads, distribution or arrangement of the resulting data among threads.  
     163 
     164%The most representative functions of the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. 
     165 
     166For example, if we want to perform a broadcast operation, only 2 steps are needed (\textit{c.f.} Figure \ref{fig:bcast}). Firstly, the root  
     167thread, along with the master threads of other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the  
     168master threads send data to other threads via local memory transfer.  
     169 
     170\begin{figure}[ht] 
     171\centering 
     172\includegraphics[scale=0.3]{bcast.png}  
     173\caption{} 
     174\label{fig:bcast} 
     175\end{figure} 
     176 
     177Figure \ref{fig:allreduce} illustrates how the \verb|MPI_Allreduce| function is proceeded in the endpoint interface. First of all, We  
     178perform a intra-process ``allreduce'' operation: source data is reduced from slave threads to the master thread via local memory transfer.  
     179Next, alm master threads call the classic \verb|MPI_Allreduce| routine. Finally, all master threads send the updated reduced data to its  
     180slaves via local memory transfer.  
     181 
     182\begin{figure}[ht] 
     183\centering 
     184\includegraphics[scale=0.3]{allreduce.png}  
     185\caption{} 
     186\label{fig:allreduce} 
     187\end{figure} 
     188 
     189Other MPI routines, such as \verb|MPI_Wait|, \verb|MPI_Intercomm_create| \textit{etc.}, can be found in the technique report of the  
     190endpoint interface. 
     191 
     192\section{The multi-threaded XIOS and performce results} 
     193 
     194The development of endpoint interface for thread-friendly XIOS library took about one year and a half. The main difficulty is the  
     195co-existance of MPI processes and OpenMP threads. All MPI classes must be redefined in the endpoint interface along with all the routines.  
     196The development is now available on the forge server: \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp}. One  
     197technique report is also available in which one can find more detail about how endpoint works and how the routines are implemented  
     198\cite{ep:2018}. We must note that the thread-friendly XIOS library is still in the phase of optimization. It will be released in the  
     199future with a stable version. 
     200 
     201All the funcionalities of XIOS is reserved in its thread-friendly version. Single threaded code can work successfully with the new  
     202version of XIOS. For multi-threaded models, some modifications are needed in order to work with the multi-threaded XIOS library. Detail can  
     203be found in our technique report \cite{ep:2018}. 
     204 
     205Even though the multi-threaded 
    105206 
    106207\section{Performance of LMDZ using EP\_XIOS} 
     
    114215simulation duration settings: 1 day, 5 days, 15 days, and 31 days.  
    115216 
    116 \begin{figure}[h] 
     217\begin{figure}[ht] 
    117218 \centering 
    118219 \includegraphics[scale = 0.6]{LMDZ_perf.png} 
     
    131232decrease in time of 25\%. Even the 25\% may seems to be small, it is still a gain in performance with existing computing resources. 
    132233 
     234\section{Performance of EP\_XIOS} 
     235 
     236workfloz\_cmip6  
     237light output 
     23824*8+2 
     23930s - 52s 
     24032 days 
     241histmth with daily output 
     242 
    133243\section{Perspectives of EP\_XIOS} 
    134244 
     245 
     246\bibliographystyle{plain} 
     247\bibliography{reference} 
     248 
    135249\end{document}           
  • XIOS/dev/branch_openmp/Note/reference.bib

    r1548 r1552  
    66 
    77@conference {Dinan:2013, 
    8  title = {Enabling MPI Interoperability Through Flexible Communication Endpoints}, 
     8 title = {Enabling {MPI} {I}nteroperability {T}hrough {F}lexible {C}ommunication {E}ndpoints}, 
    99 booktitle = {EuroMPI 2013}, 
    1010 year = {2013}, 
     
    1616 author={S. Sridharan and J. Dinan and D. D. Kalamkar}, 
    1717 booktitle={SC14: International Conference for High Performance Computing, Networking, Storage and Analysis}, 
    18  title={Enabling Efficient Multithreaded MPI Communication through a Library-Based Implementation of MPI Endpoints}, 
     18 title={Enabling {E}fficient {M}ultithreaded {MPI} {C}ommunication {T}hrough a {L}ibrary-{B}ased {I}mplementation of {MPI} {E}ndpoints}, 
    1919 year={2014}, 
    2020 pages={487-498}, 
    2121 month={Nov} 
    2222} 
     23 
     24@techreport{ep:2018, 
     25 author={XIOS developper group}, 
     26 title={Note for {XIOS} {E}ndpoints}, 
     27 year={2018}, 
     28 address={\url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp/Note}} 
     29} 
  • XIOS/dev/branch_openmp/Note/tag.svg

    r1548 r1552  
    55    <defs/> 
    66    <g> 
    7         <g latexdraw:type="rectangle" id="id480"> 
     7        <g latexdraw:type="rectangle" id="id32"> 
    88            <rect x="661.0" y="201.0" width="178.0" height="38.0" 
    99                latexdraw:borderPos="outer" stroke-width="2.0" 
    1010                stroke="black" fill="#f0d7e6"/> 
    1111        </g> 
    12         <g latexdraw:type="rectangle" id="id481"> 
     12        <g latexdraw:type="rectangle" id="id33"> 
    1313            <rect x="221.0" y="201.0" width="258.0" height="38.0" 
    1414                latexdraw:borderPos="outer" stroke-width="2.0" 
    1515                stroke="black" fill="#d0f7e0"/> 
    1616        </g> 
    17         <g latexdraw:type="rectangle" id="id482"> 
     17        <g latexdraw:type="rectangle" id="id34"> 
    1818            <rect x="481.0" y="201.0" width="178.0" height="38.0" 
    1919                latexdraw:borderPos="outer" stroke-width="2.0" 
    2020                stroke="black" fill="#c8e7ec"/> 
    2121        </g> 
    22         <g latexdraw:type="rectangle" id="id483"> 
     22        <g latexdraw:type="rectangle" id="id35"> 
    2323            <rect x="221.0" y="201.0" width="618.0" height="38.0" 
    2424                latexdraw:borderPos="outer" stroke-width="2.0" 
    2525                stroke="black" fill="none"/> 
    2626        </g> 
    27         <g latexdraw:type="line" id="id484"> 
     27        <g latexdraw:type="line" id="id36"> 
    2828            <line x1="480.0" x2="480.0" y1="200.0" y2="240.0" 
    2929                stroke-width="2.0" stroke="black" latexdraw:rotationAngle="0.0"/> 
    3030        </g> 
    31         <g latexdraw:type="line" id="id485"> 
     31        <g latexdraw:type="line" id="id37"> 
    3232            <line x1="660.0" x2="660.0" y1="200.0" y2="240.0" 
    3333                stroke-width="2.0" stroke="black" latexdraw:rotationAngle="0.0"/> 
    3434        </g> 
    35         <g latexdraw:type="text" id="id486" font-family="Dialog" 
     35        <g latexdraw:type="text" id="id38" font-family="Dialog" 
    3636            fill="black" font-size="18" latexdraw:x="320.0"> 
    37             <text x="320.0" y="230.0"><![CDATA[EP_tag]]></text> 
     37            <text x="320.0" y="230.0"><![CDATA[Tag]]></text> 
    3838        </g> 
    39         <g latexdraw:type="text" id="id489" font-family="Dialog" 
     39        <g latexdraw:type="text" id="id41" font-family="Dialog" 
    4040            fill="black" font-size="18" latexdraw:x="510.0"> 
    4141            <text x="510.0" y="230.0"><![CDATA[src local rank]]></text> 
    4242        </g> 
    43         <g latexdraw:type="text" id="id492" font-family="Dialog" 
     43        <g latexdraw:type="text" id="id43" font-family="Dialog" 
    4444            fill="black" font-size="18" latexdraw:x="690.0"> 
    4545            <text x="690.0" y="230.0"><![CDATA[dst local rank]]></text> 
    4646        </g> 
    47         <g latexdraw:type="text" id="id495" font-family="Dialog" 
     47        <g latexdraw:type="text" id="id45" font-family="Dialog" 
    4848            fill="black" font-size="18" latexdraw:x="100.0"> 
    4949            <text x="100.0" y="230.0"><![CDATA[MPI_tag =]]></text> 
    5050        </g> 
    51         <g latexdraw:type="text" id="id498" font-family="Dialog" 
     51        <g latexdraw:type="text" id="id47" font-family="Dialog" 
    5252            fill="black" font-size="18" latexdraw:x="400.0"> 
    5353            <text x="400.0" y="270.0"><![CDATA[15 bits]]></text> 
    5454        </g> 
    55         <g latexdraw:type="text" id="id501" font-family="Dialog" 
     55        <g latexdraw:type="text" id="id49" font-family="Dialog" 
    5656            fill="black" font-size="18" latexdraw:x="590.0"> 
    5757            <text x="590.0" y="270.0"><![CDATA[8 bits]]></text> 
    5858        </g> 
    59         <g latexdraw:type="text" id="id504" font-family="Dialog" 
     59        <g latexdraw:type="text" id="id51" font-family="Dialog" 
    6060            fill="black" font-size="18" latexdraw:x="770.0"> 
    6161            <text x="770.0" y="270.0"><![CDATA[8 bits]]></text> 
    6262        </g> 
    63         <g latexdraw:type="text" id="id507" font-family="Dialog" 
     63        <g latexdraw:type="text" id="id53" font-family="Dialog" 
    6464            fill="black" font-size="18" latexdraw:x="570.0"> 
    65             <text x="570.0" y="290.0"><![CDATA[0 ~ 256]]></text> 
     65            <text x="570.0" y="290.0"><![CDATA[0 ~ 255]]></text> 
    6666        </g> 
    67         <g latexdraw:type="text" id="id510" font-family="Dialog" 
     67        <g latexdraw:type="text" id="id56" font-family="Dialog" 
    6868            fill="black" font-size="18" latexdraw:x="750.0"> 
    69             <text x="750.0" y="290.0"><![CDATA[0 ~ 256]]></text> 
     69            <text x="750.0" y="290.0"><![CDATA[0 ~ 255]]></text> 
    7070        </g> 
    71         <g latexdraw:type="text" id="id513" font-family="Dialog" 
     71        <g latexdraw:type="text" id="id59" font-family="Dialog" 
    7272            fill="black" font-size="18" latexdraw:x="370.0"> 
    73             <text x="370.0" y="290.0"><![CDATA[0 ~ 32768]]></text> 
     73            <text x="370.0" y="290.0"><![CDATA[0 ~ 32767]]></text> 
    7474        </g> 
    7575    </g> 
     
    8989            <latexdraw:version>2.0.8</latexdraw:version> 
    9090            <latexdraw:size> 
    91                 <latexdraw:width>1867</latexdraw:width> 
    92                 <latexdraw:height>1176</latexdraw:height> 
     91                <latexdraw:width>2507</latexdraw:width> 
     92                <latexdraw:height>1576</latexdraw:height> 
    9393            </latexdraw:size> 
    9494            <latexdraw:position> 
     
    9696                <latexdraw:y>24</latexdraw:y> 
    9797            </latexdraw:position> 
    98             <latexdraw:dividerPosition>0.8489555436529191</latexdraw:dividerPosition> 
     98            <latexdraw:dividerPosition>0.9976067012365377</latexdraw:dividerPosition> 
    9999            <latexdraw:displayCodePanel>true</latexdraw:displayCodePanel> 
    100100            <latexdraw:displayXScale>true</latexdraw:displayXScale> 
Note: See TracChangeset for help on using the changeset viewer.