- Timestamp:
- 06/27/18 15:18:29 (6 years ago)
- Location:
- XIOS/dev/branch_openmp/Note
- Files:
-
- 16 edited
Legend:
- Unmodified
- Added
- Removed
-
XIOS/dev/branch_openmp/Note/EP note.aux
r1548 r1552 21 21 \@writefile{toc}{\contentsline {section}{\numberline {6}Inter-communicator}{8}} 22 22 \@writefile{toc}{\contentsline {subsection}{\numberline {6.1}The splitting of intra-communicator}{8}} 23 \@writefile{toc}{\contentsline {subsection}{\numberline {6.2}The creation of inter-communicator}{ 10}}23 \@writefile{toc}{\contentsline {subsection}{\numberline {6.2}The creation of inter-communicator}{9}} 24 24 \citation{MPI} 25 25 \@writefile{toc}{\contentsline {subsection}{\numberline {6.3}The merge of inter-communicators}{12}} -
XIOS/dev/branch_openmp/Note/EP note.log
r1548 r1552 1 This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24) 2 2 JUN 2018 16:191 This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24) 26 JUN 2018 15:00 2 2 entering extended mode 3 3 restricted \write18 enabled. … … 638 638 639 639 640 Overfull \hbox ( 25.85847pt too wide) in paragraph at lines 239--240640 Overfull \hbox (44.74739pt too wide) in paragraph at lines 239--240 641 641 []\OT1/cmr/m/n/10 We call []\OT1/cmtt/m/n/10 MPI_Bcast(buffer, count, datatype, 642 mpi_root = 1, mpi_comm) \OT1/cmr/m/n/10 .643 [] 644 645 <bcast.png, id=36, 743.77875pt x 552.0625pt>642 mpi_root = 1, mpi_comm) \OT1/cmr/m/n/10 with 643 [] 644 645 <bcast.png, id=36, 512.91624pt x 552.0625pt> 646 646 File: bcast.png Graphic file (type png) 647 647 <use bcast.png> 648 648 Package pdftex.def Info: bcast.png used on input line 244. 649 (pdftex.def) Requested size: 223.13535pt x 165.62003pt.649 (pdftex.def) Requested size: 153.87605pt x 165.62003pt. 650 650 651 651 Overfull \hbox (16.20544pt too wide) in paragraph at lines 249--251 … … 687 687 Package pdftex.def Info: intercomm.png used on input line 344. 688 688 (pdftex.def) Requested size: 311.06451pt x 137.9163pt. 689 [10 <./intercomm.png (PNG copy)>] 690 <intercomm_step.png, id=5 6, 1309.89375pt x 683.55376pt>689 690 <intercomm_step.png, id=53, 1309.89375pt x 683.55376pt> 691 691 File: intercomm_step.png Graphic file (type png) 692 692 … … 694 694 Package pdftex.def Info: intercomm_step.png used on input line 369. 695 695 (pdftex.def) Requested size: 327.47264pt x 170.88802pt. 696 696 [10 <./intercomm.png (PNG copy)>] 697 697 Overfull \hbox (6.07089pt too wide) in paragraph at lines 372--375 698 698 \OT1/cmr/m/n/10 should add an \OT1/cmr/m/it/10 pri-or-ity check \OT1/cmr/m/n/10 … … 827 827 xlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texlive/ 828 828 texmf-dist/fonts/type1/urw/times/utmr8a.pfb> 829 Output written on "EP note.pdf" (17 pages, 10 91602bytes).829 Output written on "EP note.pdf" (17 pages, 1089153 bytes). 830 830 PDF statistics: 831 831 133 PDF objects out of 1000 (max. 8388607) -
XIOS/dev/branch_openmp/Note/EP note.tex
r1548 r1552 236 236 237 237 \begin{itemize} 238 \item[1.] EP process with rank 4 send the buffer to EP process rank 3 which is a master thread.239 \item[ 2.] We call \verb|MPI_Bcast(buffer, count, datatype, mpi_root = 1, mpi_comm) |.240 \item[ 3.] All master threadssend the buffer to its slaves.238 %\item[1.] EP process with rank 4 send the buffer to EP process rank 3 which is a master thread. 239 \item[1.] We call \verb|MPI_Bcast(buffer, count, datatype, mpi_root = 1, mpi_comm) | with EP processes rank 0, 4, 6, and 9. 240 \item[2.] EP processes rank 0, 4, 6, and 9 send the buffer to its slaves. 241 241 \end{itemize} 242 242 -
XIOS/dev/branch_openmp/Note/bcast.svg
r1548 r1552 4 4 version="1.1" baseProfile="full"> 5 5 <defs> 6 <marker overflow="visible" orient="auto" id="arrow2- 3386">6 <marker overflow="visible" orient="auto" id="arrow2-619"> 7 7 <path 8 8 d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " 9 9 latexdraw:arrSizeNum="3.0" fill="black"/> 10 10 </marker> 11 <marker overflow="visible" orient="auto" id="arrow2-3391"> 12 <path 13 d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " 14 latexdraw:arrSizeNum="3.0" fill="black"/> 15 </marker> 16 <marker overflow="visible" orient="auto" id="arrow2-3442"> 11 <marker overflow="visible" orient="auto" id="arrow2-657"> 17 12 <path 18 13 d="M 0.0 0.0 L -4.8173611111111105 1.720486111111111 L -2.890416666666666 0.0 L -4.8173611111111105 -1.720486111111111 z " … … 21 16 </defs> 22 17 <g> 23 <g latexdraw:type="drawing" id="id3206"> 24 <g latexdraw:type="drawing" id="id3207"> 25 <g latexdraw:type="drawing" id="id3208"> 26 <g latexdraw:type="rectangle" id="id3209"> 27 <rect x="351.0" y="531.0" width="38.0" 28 height="118.0" latexdraw:borderPos="outer" 29 stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 30 </g> 31 <g latexdraw:type="rectangle" id="id3210"> 32 <rect x="351.0" y="391.0" width="38.0" 33 height="118.0" latexdraw:borderPos="outer" 34 stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 35 </g> 36 <g latexdraw:type="rectangle" id="id3211"> 37 <rect x="351.0" y="251.0" width="38.0" 38 height="118.0" latexdraw:borderPos="outer" 39 stroke-width="2.0" stroke="black" fill="#f1daf9"/> 40 </g> 41 <g latexdraw:type="rectangle" id="id3212"> 42 <rect x="351.0" y="111.0" width="38.0" 43 height="118.0" latexdraw:borderPos="outer" 44 stroke-width="2.0" stroke="black" fill="#fafac9"/> 45 </g> 46 </g> 47 <g latexdraw:type="drawing" id="id3214"> 48 <g latexdraw:type="square" id="id3215"> 49 <rect x="361.0" y="121.0" width="18.0" 50 height="18.0" latexdraw:borderPos="outer" 51 stroke-width="2.0" stroke="black" fill="white"/> 52 </g> 53 <g latexdraw:type="square" id="id3216"> 54 <rect x="361.0" y="161.0" width="18.0" 55 height="18.0" latexdraw:borderPos="outer" 56 stroke-width="2.0" stroke="black" fill="white"/> 57 </g> 58 <g latexdraw:type="square" id="id3217"> 59 <rect x="361.0" y="201.0" width="18.0" 60 height="18.0" latexdraw:borderPos="outer" 61 stroke-width="2.0" stroke="black" fill="white"/> 62 </g> 63 </g> 64 <g latexdraw:type="square" id="id3219"> 65 <rect x="361.0" y="261.0" width="18.0" height="18.0" 66 latexdraw:borderPos="outer" stroke-width="2.0" 67 stroke="black" fill="white"/> 68 </g> 69 <g latexdraw:type="square" id="id3220"> 70 <rect x="361.0" y="301.0" width="18.0" height="18.0" 71 latexdraw:borderPos="outer" stroke-width="2.0" 72 stroke="black" fill="red"/> 73 </g> 74 <g latexdraw:type="square" id="id3221"> 75 <rect x="361.0" y="341.0" width="18.0" height="18.0" 76 latexdraw:borderPos="outer" stroke-width="2.0" 77 stroke="black" fill="white"/> 78 </g> 79 <g latexdraw:type="drawing" id="id3222"> 80 <g latexdraw:type="square" id="id3223"> 81 <rect x="361.0" y="401.0" width="18.0" 82 height="18.0" latexdraw:borderPos="outer" 83 stroke-width="2.0" stroke="black" fill="white"/> 84 </g> 85 <g latexdraw:type="square" id="id3224"> 86 <rect x="361.0" y="441.0" width="18.0" 87 height="18.0" latexdraw:borderPos="outer" 88 stroke-width="2.0" stroke="black" fill="white"/> 89 </g> 90 <g latexdraw:type="square" id="id3225"> 91 <rect x="361.0" y="481.0" width="18.0" 92 height="18.0" latexdraw:borderPos="outer" 93 stroke-width="2.0" stroke="black" fill="white"/> 94 </g> 95 </g> 96 <g latexdraw:type="drawing" id="id3227"> 97 <g latexdraw:type="square" id="id3228"> 98 <rect x="361.0" y="541.0" width="18.0" 99 height="18.0" latexdraw:borderPos="outer" 100 stroke-width="2.0" stroke="black" fill="white"/> 101 </g> 102 <g latexdraw:type="square" id="id3229"> 103 <rect x="361.0" y="581.0" width="18.0" 104 height="18.0" latexdraw:borderPos="outer" 105 stroke-width="2.0" stroke="black" fill="white"/> 106 </g> 107 <g latexdraw:type="square" id="id3230"> 108 <rect x="361.0" y="621.0" width="18.0" 109 height="18.0" latexdraw:borderPos="outer" 110 stroke-width="2.0" stroke="black" fill="white"/> 111 </g> 112 </g> 113 <g latexdraw:type="text" id="id3232" 114 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 115 <text x="320.0" y="140.0"><![CDATA[0]]></text> 116 </g> 117 <g latexdraw:type="text" id="id3235" 118 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 119 <text x="320.0" y="180.0"><![CDATA[1]]></text> 120 </g> 121 <g latexdraw:type="text" id="id3238" 122 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 123 <text x="320.0" y="220.0"><![CDATA[2]]></text> 124 </g> 125 <g latexdraw:type="text" id="id3241" 126 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 127 <text x="320.0" y="280.0"><![CDATA[3]]></text> 128 </g> 129 <g latexdraw:type="text" id="id3244" 130 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 131 <text x="320.0" y="320.0"><![CDATA[4]]></text> 132 </g> 133 <g latexdraw:type="text" id="id3247" 134 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 135 <text x="320.0" y="360.0"><![CDATA[5]]></text> 136 </g> 137 <g latexdraw:type="text" id="id3250" 138 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 139 <text x="320.0" y="420.0"><![CDATA[6]]></text> 140 </g> 141 <g latexdraw:type="text" id="id3253" 142 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 143 <text x="320.0" y="460.0"><![CDATA[7]]></text> 144 </g> 145 <g latexdraw:type="text" id="id3256" 146 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 147 <text x="320.0" y="500.0"><![CDATA[8]]></text> 148 </g> 149 <g latexdraw:type="text" id="id3259" 150 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 151 <text x="320.0" y="560.0"><![CDATA[9]]></text> 152 </g> 153 <g latexdraw:type="text" id="id3262" 154 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 155 <text x="320.0" y="600.0"><![CDATA[10]]></text> 156 </g> 157 <g latexdraw:type="text" id="id3265" 158 font-family="Dialog" fill="black" font-size="18" latexdraw:x="320.0"> 159 <text x="320.0" y="640.0"><![CDATA[11]]></text> 160 </g> 161 </g> 162 <g latexdraw:type="drawing" id="id3269"> 163 <g latexdraw:type="drawing" id="id3270"> 164 <g latexdraw:type="rectangle" id="id3271"> 165 <rect x="571.0" y="531.0" width="38.0" 166 height="118.0" latexdraw:borderPos="outer" 167 stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 168 </g> 169 <g latexdraw:type="rectangle" id="id3272"> 170 <rect x="571.0" y="391.0" width="38.0" 171 height="118.0" latexdraw:borderPos="outer" 172 stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 173 </g> 174 <g latexdraw:type="rectangle" id="id3273"> 175 <rect x="571.0" y="251.0" width="38.0" 176 height="118.0" latexdraw:borderPos="outer" 177 stroke-width="2.0" stroke="black" fill="#f1daf9"/> 178 </g> 179 <g latexdraw:type="rectangle" id="id3274"> 180 <rect x="571.0" y="111.0" width="38.0" 181 height="118.0" latexdraw:borderPos="outer" 182 stroke-width="2.0" stroke="black" fill="#fafac9"/> 183 </g> 184 </g> 185 <g latexdraw:type="drawing" id="id3276"> 186 <g latexdraw:type="square" id="id3277"> 187 <rect x="581.0" y="121.0" width="18.0" 188 height="18.0" latexdraw:borderPos="outer" 189 stroke-width="2.0" stroke="black" fill="white"/> 190 </g> 191 <g latexdraw:type="square" id="id3278"> 192 <rect x="581.0" y="161.0" width="18.0" 193 height="18.0" latexdraw:borderPos="outer" 194 stroke-width="2.0" stroke="black" fill="white"/> 195 </g> 196 <g latexdraw:type="square" id="id3279"> 197 <rect x="581.0" y="201.0" width="18.0" 198 height="18.0" latexdraw:borderPos="outer" 199 stroke-width="2.0" stroke="black" fill="white"/> 200 </g> 201 </g> 202 <g latexdraw:type="square" id="id3281"> 203 <rect x="581.0" y="261.0" width="18.0" height="18.0" 204 latexdraw:borderPos="outer" stroke-width="2.0" 205 stroke="black" fill="red"/> 206 </g> 207 <g latexdraw:type="square" id="id3282"> 208 <rect x="581.0" y="301.0" width="18.0" height="18.0" 209 latexdraw:borderPos="outer" stroke-width="2.0" 210 stroke="black" fill="red"/> 211 </g> 212 <g latexdraw:type="square" id="id3283"> 213 <rect x="581.0" y="341.0" width="18.0" height="18.0" 214 latexdraw:borderPos="outer" stroke-width="2.0" 215 stroke="black" fill="white"/> 216 </g> 217 <g latexdraw:type="drawing" id="id3284"> 218 <g latexdraw:type="square" id="id3285"> 219 <rect x="581.0" y="401.0" width="18.0" 220 height="18.0" latexdraw:borderPos="outer" 221 stroke-width="2.0" stroke="black" fill="white"/> 222 </g> 223 <g latexdraw:type="square" id="id3286"> 224 <rect x="581.0" y="441.0" width="18.0" 225 height="18.0" latexdraw:borderPos="outer" 226 stroke-width="2.0" stroke="black" fill="white"/> 227 </g> 228 <g latexdraw:type="square" id="id3287"> 229 <rect x="581.0" y="481.0" width="18.0" 230 height="18.0" latexdraw:borderPos="outer" 231 stroke-width="2.0" stroke="black" fill="white"/> 232 </g> 233 </g> 234 <g latexdraw:type="drawing" id="id3289"> 235 <g latexdraw:type="square" id="id3290"> 236 <rect x="581.0" y="541.0" width="18.0" 237 height="18.0" latexdraw:borderPos="outer" 238 stroke-width="2.0" stroke="black" fill="white"/> 239 </g> 240 <g latexdraw:type="square" id="id3291"> 241 <rect x="581.0" y="581.0" width="18.0" 242 height="18.0" latexdraw:borderPos="outer" 243 stroke-width="2.0" stroke="black" fill="white"/> 244 </g> 245 <g latexdraw:type="square" id="id3292"> 246 <rect x="581.0" y="621.0" width="18.0" 247 height="18.0" latexdraw:borderPos="outer" 248 stroke-width="2.0" stroke="black" fill="white"/> 249 </g> 250 </g> 251 <g latexdraw:type="text" id="id3294" 252 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 253 <text x="540.0" y="140.0"><![CDATA[0]]></text> 254 </g> 255 <g latexdraw:type="text" id="id3297" 256 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 257 <text x="540.0" y="180.0"><![CDATA[1]]></text> 258 </g> 259 <g latexdraw:type="text" id="id3300" 260 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 261 <text x="540.0" y="220.0"><![CDATA[2]]></text> 262 </g> 263 <g latexdraw:type="text" id="id3303" 264 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 265 <text x="540.0" y="280.0"><![CDATA[3]]></text> 266 </g> 267 <g latexdraw:type="text" id="id3306" 268 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 269 <text x="540.0" y="320.0"><![CDATA[4]]></text> 270 </g> 271 <g latexdraw:type="text" id="id3309" 272 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 273 <text x="540.0" y="360.0"><![CDATA[5]]></text> 274 </g> 275 <g latexdraw:type="text" id="id3312" 276 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 277 <text x="540.0" y="420.0"><![CDATA[6]]></text> 278 </g> 279 <g latexdraw:type="text" id="id3315" 280 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 281 <text x="540.0" y="460.0"><![CDATA[7]]></text> 282 </g> 283 <g latexdraw:type="text" id="id3318" 284 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 285 <text x="540.0" y="500.0"><![CDATA[8]]></text> 286 </g> 287 <g latexdraw:type="text" id="id3321" 288 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 289 <text x="540.0" y="560.0"><![CDATA[9]]></text> 290 </g> 291 <g latexdraw:type="text" id="id3324" 292 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 293 <text x="540.0" y="600.0"><![CDATA[10]]></text> 294 </g> 295 <g latexdraw:type="text" id="id3327" 296 font-family="Dialog" fill="black" font-size="18" latexdraw:x="540.0"> 297 <text x="540.0" y="640.0"><![CDATA[11]]></text> 298 </g> 299 </g> 300 <g latexdraw:type="drawing" id="id3331"> 301 <g latexdraw:type="drawing" id="id3332"> 302 <g latexdraw:type="rectangle" id="id3333"> 303 <rect x="801.0" y="531.0" width="38.0" 304 height="118.0" latexdraw:borderPos="outer" 305 stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 306 </g> 307 <g latexdraw:type="rectangle" id="id3334"> 308 <rect x="801.0" y="391.0" width="38.0" 309 height="118.0" latexdraw:borderPos="outer" 310 stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 311 </g> 312 <g latexdraw:type="rectangle" id="id3335"> 313 <rect x="801.0" y="251.0" width="38.0" 314 height="118.0" latexdraw:borderPos="outer" 315 stroke-width="2.0" stroke="black" fill="#f1daf9"/> 316 </g> 317 <g latexdraw:type="rectangle" id="id3336"> 318 <rect x="801.0" y="111.0" width="38.0" 319 height="118.0" latexdraw:borderPos="outer" 320 stroke-width="2.0" stroke="black" fill="#fafac9"/> 321 </g> 322 </g> 323 <g latexdraw:type="square" id="id3338"> 324 <rect x="811.0" y="121.0" width="18.0" height="18.0" 325 latexdraw:borderPos="outer" stroke-width="2.0" 326 stroke="black" fill="red"/> 327 </g> 328 <g latexdraw:type="square" id="id3339"> 329 <rect x="811.0" y="161.0" width="18.0" height="18.0" 330 latexdraw:borderPos="outer" stroke-width="2.0" 331 stroke="black" fill="white"/> 332 </g> 333 <g latexdraw:type="square" id="id3340"> 334 <rect x="811.0" y="201.0" width="18.0" height="18.0" 335 latexdraw:borderPos="outer" stroke-width="2.0" 336 stroke="black" fill="white"/> 337 </g> 338 <g latexdraw:type="square" id="id3341"> 339 <rect x="811.0" y="261.0" width="18.0" height="18.0" 340 latexdraw:borderPos="outer" stroke-width="2.0" 341 stroke="black" fill="red"/> 342 </g> 343 <g latexdraw:type="square" id="id3342"> 344 <rect x="811.0" y="301.0" width="18.0" height="18.0" 345 latexdraw:borderPos="outer" stroke-width="2.0" 346 stroke="black" fill="red"/> 347 </g> 348 <g latexdraw:type="square" id="id3343"> 349 <rect x="811.0" y="341.0" width="18.0" height="18.0" 350 latexdraw:borderPos="outer" stroke-width="2.0" 351 stroke="black" fill="white"/> 352 </g> 353 <g latexdraw:type="square" id="id3344"> 354 <rect x="811.0" y="401.0" width="18.0" height="18.0" 355 latexdraw:borderPos="outer" stroke-width="2.0" 356 stroke="black" fill="red"/> 357 </g> 358 <g latexdraw:type="square" id="id3345"> 359 <rect x="811.0" y="441.0" width="18.0" height="18.0" 360 latexdraw:borderPos="outer" stroke-width="2.0" 361 stroke="black" fill="white"/> 362 </g> 363 <g latexdraw:type="square" id="id3346"> 364 <rect x="811.0" y="481.0" width="18.0" height="18.0" 365 latexdraw:borderPos="outer" stroke-width="2.0" 366 stroke="black" fill="white"/> 367 </g> 368 <g latexdraw:type="square" id="id3347"> 369 <rect x="811.0" y="541.0" width="18.0" height="18.0" 370 latexdraw:borderPos="outer" stroke-width="2.0" 371 stroke="black" fill="red"/> 372 </g> 373 <g latexdraw:type="square" id="id3348"> 374 <rect x="811.0" y="581.0" width="18.0" height="18.0" 375 latexdraw:borderPos="outer" stroke-width="2.0" 376 stroke="black" fill="white"/> 377 </g> 378 <g latexdraw:type="square" id="id3349"> 379 <rect x="811.0" y="621.0" width="18.0" height="18.0" 380 latexdraw:borderPos="outer" stroke-width="2.0" 381 stroke="black" fill="white"/> 382 </g> 383 <g latexdraw:type="text" id="id3350" 384 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 385 <text x="770.0" y="140.0"><![CDATA[0]]></text> 386 </g> 387 <g latexdraw:type="text" id="id3353" 388 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 389 <text x="770.0" y="180.0"><![CDATA[1]]></text> 390 </g> 391 <g latexdraw:type="text" id="id3356" 392 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 393 <text x="770.0" y="220.0"><![CDATA[2]]></text> 394 </g> 395 <g latexdraw:type="text" id="id3359" 396 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 397 <text x="770.0" y="280.0"><![CDATA[3]]></text> 398 </g> 399 <g latexdraw:type="text" id="id3362" 400 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 401 <text x="770.0" y="320.0"><![CDATA[4]]></text> 402 </g> 403 <g latexdraw:type="text" id="id3365" 404 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 405 <text x="770.0" y="360.0"><![CDATA[5]]></text> 406 </g> 407 <g latexdraw:type="text" id="id3368" 408 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 409 <text x="770.0" y="420.0"><![CDATA[6]]></text> 410 </g> 411 <g latexdraw:type="text" id="id3371" 412 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 413 <text x="770.0" y="460.0"><![CDATA[7]]></text> 414 </g> 415 <g latexdraw:type="text" id="id3374" 416 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 417 <text x="770.0" y="500.0"><![CDATA[8]]></text> 418 </g> 419 <g latexdraw:type="text" id="id3377" 420 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 421 <text x="770.0" y="560.0"><![CDATA[9]]></text> 422 </g> 423 <g latexdraw:type="text" id="id3380" 424 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 425 <text x="770.0" y="600.0"><![CDATA[10]]></text> 426 </g> 427 <g latexdraw:type="text" id="id3383" 428 font-family="Dialog" fill="black" font-size="18" latexdraw:x="770.0"> 429 <text x="770.0" y="640.0"><![CDATA[11]]></text> 430 </g> 431 <g latexdraw:type="line" id="id3386"> 432 <line x1="640.0" x2="760.0" y1="380.0" y2="380.0" 433 stroke-width="6.0" stroke="black" 434 latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3386)"/> 435 </g> 436 <g latexdraw:type="text" id="id3387" 437 font-family="Dialog" fill="black" font-size="18" latexdraw:x="670.0"> 438 <text x="670.0" y="360.0"><![CDATA[step 2]]></text> 439 </g> 440 </g> 441 <g latexdraw:type="line" id="id3391"> 442 <line x1="410.0" x2="530.0" y1="380.0" y2="380.0" 443 stroke-width="6.0" stroke="black" 444 latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3391)"/> 445 </g> 446 <g latexdraw:type="text" id="id3392" font-family="Dialog" 447 fill="black" font-size="18" latexdraw:x="440.0"> 448 <text x="440.0" y="360.0"><![CDATA[step 1]]></text> 449 </g> 450 <g latexdraw:type="drawing" id="id3395"> 451 <g latexdraw:type="rectangle" id="id3396"> 452 <rect x="1011.0" y="531.0" width="38.0" 18 <g latexdraw:type="drawing" id="id528"> 19 <g latexdraw:type="drawing" id="id529"> 20 <g latexdraw:type="rectangle" id="id530"> 21 <rect x="581.0" y="531.0" width="38.0" 453 22 height="118.0" latexdraw:borderPos="outer" 454 23 stroke-width="2.0" stroke="black" fill="#d4f4e0"/> 455 24 </g> 456 <g latexdraw:type="rectangle" id="id 3397">457 <rect x=" 1011.0" y="391.0" width="38.0"25 <g latexdraw:type="rectangle" id="id531"> 26 <rect x="581.0" y="391.0" width="38.0" 458 27 height="118.0" latexdraw:borderPos="outer" 459 28 stroke-width="2.0" stroke="black" fill="#dfe1f2"/> 460 29 </g> 461 <g latexdraw:type="rectangle" id="id 3398">462 <rect x=" 1011.0" y="251.0" width="38.0"30 <g latexdraw:type="rectangle" id="id532"> 31 <rect x="581.0" y="251.0" width="38.0" 463 32 height="118.0" latexdraw:borderPos="outer" 464 33 stroke-width="2.0" stroke="black" fill="#f1daf9"/> 465 34 </g> 466 <g latexdraw:type="rectangle" id="id 3399">467 <rect x=" 1011.0" y="111.0" width="38.0"35 <g latexdraw:type="rectangle" id="id533"> 36 <rect x="581.0" y="111.0" width="38.0" 468 37 height="118.0" latexdraw:borderPos="outer" 469 38 stroke-width="2.0" stroke="black" fill="#fafac9"/> 470 39 </g> 471 40 </g> 472 <g latexdraw:type="drawing" id="id3401"> 473 <g latexdraw:type="square" id="id3402"> 474 <rect x="1021.0" y="261.0" width="18.0" 475 height="18.0" latexdraw:borderPos="outer" 476 stroke-width="2.0" stroke="black" fill="red"/> 477 </g> 478 <g latexdraw:type="square" id="id3403"> 479 <rect x="1021.0" y="301.0" width="18.0" 480 height="18.0" latexdraw:borderPos="outer" 481 stroke-width="2.0" stroke="black" fill="red"/> 482 </g> 483 <g latexdraw:type="square" id="id3404"> 484 <rect x="1021.0" y="341.0" width="18.0" 485 height="18.0" latexdraw:borderPos="outer" 486 stroke-width="2.0" stroke="black" fill="red"/> 487 </g> 488 </g> 489 <g latexdraw:type="text" id="id3406" font-family="Dialog" 490 fill="black" font-size="18" latexdraw:x="980.0"> 491 <text x="980.0" y="140.0"><![CDATA[0]]></text> 492 </g> 493 <g latexdraw:type="text" id="id3409" font-family="Dialog" 494 fill="black" font-size="18" latexdraw:x="980.0"> 495 <text x="980.0" y="180.0"><![CDATA[1]]></text> 496 </g> 497 <g latexdraw:type="text" id="id3412" font-family="Dialog" 498 fill="black" font-size="18" latexdraw:x="980.0"> 499 <text x="980.0" y="220.0"><![CDATA[2]]></text> 500 </g> 501 <g latexdraw:type="text" id="id3415" font-family="Dialog" 502 fill="black" font-size="18" latexdraw:x="980.0"> 503 <text x="980.0" y="280.0"><![CDATA[3]]></text> 504 </g> 505 <g latexdraw:type="text" id="id3418" font-family="Dialog" 506 fill="black" font-size="18" latexdraw:x="980.0"> 507 <text x="980.0" y="320.0"><![CDATA[4]]></text> 508 </g> 509 <g latexdraw:type="text" id="id3421" font-family="Dialog" 510 fill="black" font-size="18" latexdraw:x="980.0"> 511 <text x="980.0" y="360.0"><![CDATA[5]]></text> 512 </g> 513 <g latexdraw:type="text" id="id3424" font-family="Dialog" 514 fill="black" font-size="18" latexdraw:x="980.0"> 515 <text x="980.0" y="420.0"><![CDATA[6]]></text> 516 </g> 517 <g latexdraw:type="text" id="id3427" font-family="Dialog" 518 fill="black" font-size="18" latexdraw:x="980.0"> 519 <text x="980.0" y="460.0"><![CDATA[7]]></text> 520 </g> 521 <g latexdraw:type="text" id="id3430" font-family="Dialog" 522 fill="black" font-size="18" latexdraw:x="980.0"> 523 <text x="980.0" y="500.0"><![CDATA[8]]></text> 524 </g> 525 <g latexdraw:type="text" id="id3433" font-family="Dialog" 526 fill="black" font-size="18" latexdraw:x="980.0"> 527 <text x="980.0" y="560.0"><![CDATA[9]]></text> 528 </g> 529 <g latexdraw:type="text" id="id3436" font-family="Dialog" 530 fill="black" font-size="18" latexdraw:x="980.0"> 531 <text x="980.0" y="600.0"><![CDATA[10]]></text> 532 </g> 533 <g latexdraw:type="text" id="id3439" font-family="Dialog" 534 fill="black" font-size="18" latexdraw:x="980.0"> 535 <text x="980.0" y="640.0"><![CDATA[11]]></text> 536 </g> 537 <g latexdraw:type="line" id="id3442"> 538 <line x1="850.0" x2="970.0" y1="380.0" y2="380.0" 539 stroke-width="6.0" stroke="black" 540 latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-3442)"/> 541 </g> 542 <g latexdraw:type="text" id="id3443" font-family="Dialog" 543 fill="black" font-size="18" latexdraw:x="880.0"> 544 <text x="880.0" y="360.0"><![CDATA[step 3]]></text> 545 </g> 546 <g latexdraw:type="drawing" id="id3446"> 547 <g latexdraw:type="square" id="id3447"> 548 <rect x="1021.0" y="401.0" width="18.0" 549 height="18.0" latexdraw:borderPos="outer" 550 stroke-width="2.0" stroke="black" fill="red"/> 551 </g> 552 <g latexdraw:type="square" id="id3448"> 553 <rect x="1021.0" y="441.0" width="18.0" 554 height="18.0" latexdraw:borderPos="outer" 555 stroke-width="2.0" stroke="black" fill="red"/> 556 </g> 557 <g latexdraw:type="square" id="id3449"> 558 <rect x="1021.0" y="481.0" width="18.0" 559 height="18.0" latexdraw:borderPos="outer" 560 stroke-width="2.0" stroke="black" fill="red"/> 561 </g> 562 </g> 563 <g latexdraw:type="drawing" id="id3451"> 564 <g latexdraw:type="square" id="id3452"> 565 <rect x="1021.0" y="541.0" width="18.0" 566 height="18.0" latexdraw:borderPos="outer" 567 stroke-width="2.0" stroke="black" fill="red"/> 568 </g> 569 <g latexdraw:type="square" id="id3453"> 570 <rect x="1021.0" y="581.0" width="18.0" 571 height="18.0" latexdraw:borderPos="outer" 572 stroke-width="2.0" stroke="black" fill="red"/> 573 </g> 574 <g latexdraw:type="square" id="id3454"> 575 <rect x="1021.0" y="621.0" width="18.0" 576 height="18.0" latexdraw:borderPos="outer" 577 stroke-width="2.0" stroke="black" fill="red"/> 578 </g> 579 </g> 580 <g latexdraw:type="drawing" id="id3456"> 581 <g latexdraw:type="square" id="id3457"> 582 <rect x="1021.0" y="121.0" width="18.0" 583 height="18.0" latexdraw:borderPos="outer" 584 stroke-width="2.0" stroke="black" fill="red"/> 585 </g> 586 <g latexdraw:type="square" id="id3458"> 587 <rect x="1021.0" y="161.0" width="18.0" 588 height="18.0" latexdraw:borderPos="outer" 589 stroke-width="2.0" stroke="black" fill="red"/> 590 </g> 591 <g latexdraw:type="square" id="id3459"> 592 <rect x="1021.0" y="201.0" width="18.0" 593 height="18.0" latexdraw:borderPos="outer" 594 stroke-width="2.0" stroke="black" fill="red"/> 595 </g> 596 </g> 41 <g latexdraw:type="drawing" id="id535"> 42 <g latexdraw:type="square" id="id536"> 43 <rect x="591.0" y="121.0" width="18.0" height="18.0" 44 latexdraw:borderPos="outer" stroke-width="2.0" 45 stroke="black" fill="white"/> 46 </g> 47 <g latexdraw:type="square" id="id537"> 48 <rect x="591.0" y="161.0" width="18.0" height="18.0" 49 latexdraw:borderPos="outer" stroke-width="2.0" 50 stroke="black" fill="white"/> 51 </g> 52 <g latexdraw:type="square" id="id538"> 53 <rect x="591.0" y="201.0" width="18.0" height="18.0" 54 latexdraw:borderPos="outer" stroke-width="2.0" 55 stroke="black" fill="white"/> 56 </g> 57 </g> 58 <g latexdraw:type="square" id="id540"> 59 <rect x="591.0" y="261.0" width="18.0" height="18.0" 60 latexdraw:borderPos="outer" stroke-width="2.0" 61 stroke="black" fill="white"/> 62 </g> 63 <g latexdraw:type="square" id="id541"> 64 <rect x="591.0" y="301.0" width="18.0" height="18.0" 65 latexdraw:borderPos="outer" stroke-width="2.0" 66 stroke="black" fill="red"/> 67 </g> 68 <g latexdraw:type="square" id="id542"> 69 <rect x="591.0" y="341.0" width="18.0" height="18.0" 70 latexdraw:borderPos="outer" stroke-width="2.0" 71 stroke="black" fill="white"/> 72 </g> 73 <g latexdraw:type="drawing" id="id543"> 74 <g latexdraw:type="square" id="id544"> 75 <rect x="591.0" y="401.0" width="18.0" height="18.0" 76 latexdraw:borderPos="outer" stroke-width="2.0" 77 stroke="black" fill="white"/> 78 </g> 79 <g latexdraw:type="square" id="id545"> 80 <rect x="591.0" y="441.0" width="18.0" height="18.0" 81 latexdraw:borderPos="outer" stroke-width="2.0" 82 stroke="black" fill="white"/> 83 </g> 84 <g latexdraw:type="square" id="id546"> 85 <rect x="591.0" y="481.0" width="18.0" height="18.0" 86 latexdraw:borderPos="outer" stroke-width="2.0" 87 stroke="black" fill="white"/> 88 </g> 89 </g> 90 <g latexdraw:type="drawing" id="id548"> 91 <g latexdraw:type="square" id="id549"> 92 <rect x="591.0" y="541.0" width="18.0" height="18.0" 93 latexdraw:borderPos="outer" stroke-width="2.0" 94 stroke="black" fill="white"/> 95 </g> 96 <g latexdraw:type="square" id="id550"> 97 <rect x="591.0" y="581.0" width="18.0" height="18.0" 98 latexdraw:borderPos="outer" stroke-width="2.0" 99 stroke="black" fill="white"/> 100 </g> 101 <g latexdraw:type="square" id="id551"> 102 <rect x="591.0" y="621.0" width="18.0" height="18.0" 103 latexdraw:borderPos="outer" stroke-width="2.0" 104 stroke="black" fill="white"/> 105 </g> 106 </g> 107 <g latexdraw:type="text" id="id553" font-family="Dialog" 108 fill="black" font-size="18" latexdraw:x="550.0"> 109 <text x="550.0" y="140.0"><![CDATA[0]]></text> 110 </g> 111 <g latexdraw:type="text" id="id555" font-family="Dialog" 112 fill="black" font-size="18" latexdraw:x="550.0"> 113 <text x="550.0" y="180.0"><![CDATA[1]]></text> 114 </g> 115 <g latexdraw:type="text" id="id557" font-family="Dialog" 116 fill="black" font-size="18" latexdraw:x="550.0"> 117 <text x="550.0" y="220.0"><![CDATA[2]]></text> 118 </g> 119 <g latexdraw:type="text" id="id559" font-family="Dialog" 120 fill="black" font-size="18" latexdraw:x="550.0"> 121 <text x="550.0" y="280.0"><![CDATA[3]]></text> 122 </g> 123 <g latexdraw:type="text" id="id561" font-family="Dialog" 124 fill="black" font-size="18" latexdraw:x="550.0"> 125 <text x="550.0" y="320.0"><![CDATA[4]]></text> 126 </g> 127 <g latexdraw:type="text" id="id563" font-family="Dialog" 128 fill="black" font-size="18" latexdraw:x="550.0"> 129 <text x="550.0" y="360.0"><![CDATA[5]]></text> 130 </g> 131 <g latexdraw:type="text" id="id565" font-family="Dialog" 132 fill="black" font-size="18" latexdraw:x="550.0"> 133 <text x="550.0" y="420.0"><![CDATA[6]]></text> 134 </g> 135 <g latexdraw:type="text" id="id567" font-family="Dialog" 136 fill="black" font-size="18" latexdraw:x="550.0"> 137 <text x="550.0" y="460.0"><![CDATA[7]]></text> 138 </g> 139 <g latexdraw:type="text" id="id569" font-family="Dialog" 140 fill="black" font-size="18" latexdraw:x="550.0"> 141 <text x="550.0" y="500.0"><![CDATA[8]]></text> 142 </g> 143 <g latexdraw:type="text" id="id571" font-family="Dialog" 144 fill="black" font-size="18" latexdraw:x="550.0"> 145 <text x="550.0" y="560.0"><![CDATA[9]]></text> 146 </g> 147 <g latexdraw:type="text" id="id573" font-family="Dialog" 148 fill="black" font-size="18" latexdraw:x="550.0"> 149 <text x="550.0" y="600.0"><![CDATA[10]]></text> 150 </g> 151 <g latexdraw:type="text" id="id575" font-family="Dialog" 152 fill="black" font-size="18" latexdraw:x="550.0"> 153 <text x="550.0" y="640.0"><![CDATA[11]]></text> 154 </g> 155 </g> 156 <g latexdraw:type="drawing" id="id578"> 157 <g latexdraw:type="rectangle" id="id579"> 158 <rect x="801.0" y="531.0" width="38.0" height="118.0" 159 latexdraw:borderPos="outer" stroke-width="2.0" 160 stroke="black" fill="#d4f4e0"/> 161 </g> 162 <g latexdraw:type="rectangle" id="id580"> 163 <rect x="801.0" y="391.0" width="38.0" height="118.0" 164 latexdraw:borderPos="outer" stroke-width="2.0" 165 stroke="black" fill="#dfe1f2"/> 166 </g> 167 <g latexdraw:type="rectangle" id="id581"> 168 <rect x="801.0" y="251.0" width="38.0" height="118.0" 169 latexdraw:borderPos="outer" stroke-width="2.0" 170 stroke="black" fill="#f1daf9"/> 171 </g> 172 <g latexdraw:type="rectangle" id="id582"> 173 <rect x="801.0" y="111.0" width="38.0" height="118.0" 174 latexdraw:borderPos="outer" stroke-width="2.0" 175 stroke="black" fill="#fafac9"/> 176 </g> 177 </g> 178 <g latexdraw:type="square" id="id584"> 179 <rect x="811.0" y="121.0" width="18.0" height="18.0" 180 latexdraw:borderPos="outer" stroke-width="2.0" 181 stroke="black" fill="red"/> 182 </g> 183 <g latexdraw:type="square" id="id585"> 184 <rect x="811.0" y="161.0" width="18.0" height="18.0" 185 latexdraw:borderPos="outer" stroke-width="2.0" 186 stroke="black" fill="white"/> 187 </g> 188 <g latexdraw:type="square" id="id586"> 189 <rect x="811.0" y="201.0" width="18.0" height="18.0" 190 latexdraw:borderPos="outer" stroke-width="2.0" 191 stroke="black" fill="white"/> 192 </g> 193 <g latexdraw:type="square" id="id587"> 194 <rect x="811.0" y="301.0" width="18.0" height="18.0" 195 latexdraw:borderPos="outer" stroke-width="2.0" 196 stroke="black" fill="red"/> 197 </g> 198 <g latexdraw:type="square" id="id588"> 199 <rect x="811.0" y="341.0" width="18.0" height="18.0" 200 latexdraw:borderPos="outer" stroke-width="2.0" 201 stroke="black" fill="white"/> 202 </g> 203 <g latexdraw:type="square" id="id589"> 204 <rect x="811.0" y="401.0" width="18.0" height="18.0" 205 latexdraw:borderPos="outer" stroke-width="2.0" 206 stroke="black" fill="red"/> 207 </g> 208 <g latexdraw:type="square" id="id590"> 209 <rect x="811.0" y="441.0" width="18.0" height="18.0" 210 latexdraw:borderPos="outer" stroke-width="2.0" 211 stroke="black" fill="white"/> 212 </g> 213 <g latexdraw:type="square" id="id591"> 214 <rect x="811.0" y="481.0" width="18.0" height="18.0" 215 latexdraw:borderPos="outer" stroke-width="2.0" 216 stroke="black" fill="white"/> 217 </g> 218 <g latexdraw:type="square" id="id592"> 219 <rect x="811.0" y="541.0" width="18.0" height="18.0" 220 latexdraw:borderPos="outer" stroke-width="2.0" 221 stroke="black" fill="red"/> 222 </g> 223 <g latexdraw:type="square" id="id593"> 224 <rect x="811.0" y="581.0" width="18.0" height="18.0" 225 latexdraw:borderPos="outer" stroke-width="2.0" 226 stroke="black" fill="white"/> 227 </g> 228 <g latexdraw:type="square" id="id594"> 229 <rect x="811.0" y="621.0" width="18.0" height="18.0" 230 latexdraw:borderPos="outer" stroke-width="2.0" 231 stroke="black" fill="white"/> 232 </g> 233 <g latexdraw:type="text" id="id595" font-family="Dialog" 234 fill="black" font-size="18" latexdraw:x="770.0"> 235 <text x="770.0" y="140.0"><![CDATA[0]]></text> 236 </g> 237 <g latexdraw:type="text" id="id597" font-family="Dialog" 238 fill="black" font-size="18" latexdraw:x="770.0"> 239 <text x="770.0" y="180.0"><![CDATA[1]]></text> 240 </g> 241 <g latexdraw:type="text" id="id599" font-family="Dialog" 242 fill="black" font-size="18" latexdraw:x="770.0"> 243 <text x="770.0" y="220.0"><![CDATA[2]]></text> 244 </g> 245 <g latexdraw:type="text" id="id601" font-family="Dialog" 246 fill="black" font-size="18" latexdraw:x="770.0"> 247 <text x="770.0" y="280.0"><![CDATA[3]]></text> 248 </g> 249 <g latexdraw:type="text" id="id603" font-family="Dialog" 250 fill="black" font-size="18" latexdraw:x="770.0"> 251 <text x="770.0" y="320.0"><![CDATA[4]]></text> 252 </g> 253 <g latexdraw:type="text" id="id605" font-family="Dialog" 254 fill="black" font-size="18" latexdraw:x="770.0"> 255 <text x="770.0" y="360.0"><![CDATA[5]]></text> 256 </g> 257 <g latexdraw:type="text" id="id607" font-family="Dialog" 258 fill="black" font-size="18" latexdraw:x="770.0"> 259 <text x="770.0" y="420.0"><![CDATA[6]]></text> 260 </g> 261 <g latexdraw:type="text" id="id609" font-family="Dialog" 262 fill="black" font-size="18" latexdraw:x="770.0"> 263 <text x="770.0" y="460.0"><![CDATA[7]]></text> 264 </g> 265 <g latexdraw:type="text" id="id611" font-family="Dialog" 266 fill="black" font-size="18" latexdraw:x="770.0"> 267 <text x="770.0" y="500.0"><![CDATA[8]]></text> 268 </g> 269 <g latexdraw:type="text" id="id613" font-family="Dialog" 270 fill="black" font-size="18" latexdraw:x="770.0"> 271 <text x="770.0" y="560.0"><![CDATA[9]]></text> 272 </g> 273 <g latexdraw:type="text" id="id615" font-family="Dialog" 274 fill="black" font-size="18" latexdraw:x="770.0"> 275 <text x="770.0" y="600.0"><![CDATA[10]]></text> 276 </g> 277 <g latexdraw:type="text" id="id617" font-family="Dialog" 278 fill="black" font-size="18" latexdraw:x="770.0"> 279 <text x="770.0" y="640.0"><![CDATA[11]]></text> 280 </g> 281 <g latexdraw:type="line" id="id619"> 282 <line x1="640.0" x2="760.0" y1="380.0" y2="380.0" 283 stroke-width="6.0" stroke="black" 284 latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-619)"/> 285 </g> 286 <g latexdraw:type="text" id="id620" font-family="Dialog" 287 fill="black" font-size="18" latexdraw:x="670.0"> 288 <text x="670.0" y="360.0"><![CDATA[step 1]]></text> 289 </g> 290 <g latexdraw:type="drawing" id="id622"> 291 <g latexdraw:type="rectangle" id="id623"> 292 <rect x="1011.0" y="531.0" width="38.0" height="118.0" 293 latexdraw:borderPos="outer" stroke-width="2.0" 294 stroke="black" fill="#d4f4e0"/> 295 </g> 296 <g latexdraw:type="rectangle" id="id624"> 297 <rect x="1011.0" y="391.0" width="38.0" height="118.0" 298 latexdraw:borderPos="outer" stroke-width="2.0" 299 stroke="black" fill="#dfe1f2"/> 300 </g> 301 <g latexdraw:type="rectangle" id="id625"> 302 <rect x="1011.0" y="251.0" width="38.0" height="118.0" 303 latexdraw:borderPos="outer" stroke-width="2.0" 304 stroke="black" fill="#f1daf9"/> 305 </g> 306 <g latexdraw:type="rectangle" id="id626"> 307 <rect x="1011.0" y="111.0" width="38.0" height="118.0" 308 latexdraw:borderPos="outer" stroke-width="2.0" 309 stroke="black" fill="#fafac9"/> 310 </g> 311 </g> 312 <g latexdraw:type="drawing" id="id628"> 313 <g latexdraw:type="square" id="id629"> 314 <rect x="1021.0" y="261.0" width="18.0" height="18.0" 315 latexdraw:borderPos="outer" stroke-width="2.0" 316 stroke="black" fill="red"/> 317 </g> 318 <g latexdraw:type="square" id="id630"> 319 <rect x="1021.0" y="301.0" width="18.0" height="18.0" 320 latexdraw:borderPos="outer" stroke-width="2.0" 321 stroke="black" fill="red"/> 322 </g> 323 <g latexdraw:type="square" id="id631"> 324 <rect x="1021.0" y="341.0" width="18.0" height="18.0" 325 latexdraw:borderPos="outer" stroke-width="2.0" 326 stroke="black" fill="red"/> 327 </g> 328 </g> 329 <g latexdraw:type="text" id="id633" font-family="Dialog" 330 fill="black" font-size="18" latexdraw:x="980.0"> 331 <text x="980.0" y="140.0"><![CDATA[0]]></text> 332 </g> 333 <g latexdraw:type="text" id="id635" font-family="Dialog" 334 fill="black" font-size="18" latexdraw:x="980.0"> 335 <text x="980.0" y="180.0"><![CDATA[1]]></text> 336 </g> 337 <g latexdraw:type="text" id="id637" font-family="Dialog" 338 fill="black" font-size="18" latexdraw:x="980.0"> 339 <text x="980.0" y="220.0"><![CDATA[2]]></text> 340 </g> 341 <g latexdraw:type="text" id="id639" font-family="Dialog" 342 fill="black" font-size="18" latexdraw:x="980.0"> 343 <text x="980.0" y="280.0"><![CDATA[3]]></text> 344 </g> 345 <g latexdraw:type="text" id="id641" font-family="Dialog" 346 fill="black" font-size="18" latexdraw:x="980.0"> 347 <text x="980.0" y="320.0"><![CDATA[4]]></text> 348 </g> 349 <g latexdraw:type="text" id="id643" font-family="Dialog" 350 fill="black" font-size="18" latexdraw:x="980.0"> 351 <text x="980.0" y="360.0"><![CDATA[5]]></text> 352 </g> 353 <g latexdraw:type="text" id="id645" font-family="Dialog" 354 fill="black" font-size="18" latexdraw:x="980.0"> 355 <text x="980.0" y="420.0"><![CDATA[6]]></text> 356 </g> 357 <g latexdraw:type="text" id="id647" font-family="Dialog" 358 fill="black" font-size="18" latexdraw:x="980.0"> 359 <text x="980.0" y="460.0"><![CDATA[7]]></text> 360 </g> 361 <g latexdraw:type="text" id="id649" font-family="Dialog" 362 fill="black" font-size="18" latexdraw:x="980.0"> 363 <text x="980.0" y="500.0"><![CDATA[8]]></text> 364 </g> 365 <g latexdraw:type="text" id="id651" font-family="Dialog" 366 fill="black" font-size="18" latexdraw:x="980.0"> 367 <text x="980.0" y="560.0"><![CDATA[9]]></text> 368 </g> 369 <g latexdraw:type="text" id="id653" font-family="Dialog" 370 fill="black" font-size="18" latexdraw:x="980.0"> 371 <text x="980.0" y="600.0"><![CDATA[10]]></text> 372 </g> 373 <g latexdraw:type="text" id="id655" font-family="Dialog" 374 fill="black" font-size="18" latexdraw:x="980.0"> 375 <text x="980.0" y="640.0"><![CDATA[11]]></text> 376 </g> 377 <g latexdraw:type="line" id="id657"> 378 <line x1="850.0" x2="970.0" y1="380.0" y2="380.0" 379 stroke-width="6.0" stroke="black" 380 latexdraw:rotationAngle="0.0" marker-end="url(#arrow2-657)"/> 381 </g> 382 <g latexdraw:type="text" id="id658" font-family="Dialog" 383 fill="black" font-size="18" latexdraw:x="880.0"> 384 <text x="880.0" y="360.0"><![CDATA[step2]]></text> 385 </g> 386 <g latexdraw:type="drawing" id="id661"> 387 <g latexdraw:type="square" id="id662"> 388 <rect x="1021.0" y="401.0" width="18.0" height="18.0" 389 latexdraw:borderPos="outer" stroke-width="2.0" 390 stroke="black" fill="red"/> 391 </g> 392 <g latexdraw:type="square" id="id663"> 393 <rect x="1021.0" y="441.0" width="18.0" height="18.0" 394 latexdraw:borderPos="outer" stroke-width="2.0" 395 stroke="black" fill="red"/> 396 </g> 397 <g latexdraw:type="square" id="id664"> 398 <rect x="1021.0" y="481.0" width="18.0" height="18.0" 399 latexdraw:borderPos="outer" stroke-width="2.0" 400 stroke="black" fill="red"/> 401 </g> 402 </g> 403 <g latexdraw:type="drawing" id="id666"> 404 <g latexdraw:type="square" id="id667"> 405 <rect x="1021.0" y="541.0" width="18.0" height="18.0" 406 latexdraw:borderPos="outer" stroke-width="2.0" 407 stroke="black" fill="red"/> 408 </g> 409 <g latexdraw:type="square" id="id668"> 410 <rect x="1021.0" y="581.0" width="18.0" height="18.0" 411 latexdraw:borderPos="outer" stroke-width="2.0" 412 stroke="black" fill="red"/> 413 </g> 414 <g latexdraw:type="square" id="id669"> 415 <rect x="1021.0" y="621.0" width="18.0" height="18.0" 416 latexdraw:borderPos="outer" stroke-width="2.0" 417 stroke="black" fill="red"/> 418 </g> 419 </g> 420 <g latexdraw:type="drawing" id="id671"> 421 <g latexdraw:type="square" id="id672"> 422 <rect x="1021.0" y="121.0" width="18.0" height="18.0" 423 latexdraw:borderPos="outer" stroke-width="2.0" 424 stroke="black" fill="red"/> 425 </g> 426 <g latexdraw:type="square" id="id673"> 427 <rect x="1021.0" y="161.0" width="18.0" height="18.0" 428 latexdraw:borderPos="outer" stroke-width="2.0" 429 stroke="black" fill="red"/> 430 </g> 431 <g latexdraw:type="square" id="id674"> 432 <rect x="1021.0" y="201.0" width="18.0" height="18.0" 433 latexdraw:borderPos="outer" stroke-width="2.0" 434 stroke="black" fill="red"/> 435 </g> 436 </g> 437 <g latexdraw:type="square" id="id676"> 438 <rect x="811.0" y="261.0" width="18.0" height="18.0" 439 latexdraw:borderPos="outer" stroke-width="2.0" 440 stroke="black" fill="white"/> 597 441 </g> 598 442 </g> … … 612 456 <latexdraw:version>2.0.8</latexdraw:version> 613 457 <latexdraw:size> 614 <latexdraw:width> 1867</latexdraw:width>615 <latexdraw:height>1 176</latexdraw:height>458 <latexdraw:width>2507</latexdraw:width> 459 <latexdraw:height>1576</latexdraw:height> 616 460 </latexdraw:size> 617 461 <latexdraw:position> … … 619 463 <latexdraw:y>24</latexdraw:y> 620 464 </latexdraw:position> 621 <latexdraw:dividerPosition>0. 8489555436529191</latexdraw:dividerPosition>465 <latexdraw:dividerPosition>0.9976067012365377</latexdraw:dividerPosition> 622 466 <latexdraw:displayCodePanel>true</latexdraw:displayCodePanel> 623 467 <latexdraw:displayXScale>true</latexdraw:displayXScale> -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.aux
r1551 r1552 8 8 \@writefile{lof}{\contentsline {figure}{\numberline {3}{\ignorespaces }}{3}} 9 9 \newlabel{fig:omp}{{3}{3}} 10 \@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces }}{3}} 11 \newlabel{fig:scheme}{{4}{3}} 12 \@writefile{toc}{\contentsline {section}{\numberline {3}Performance of LMDZ using EP\_XIOS}{5}} 13 \@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces Speedup obtained by using EP in LMDZ simulations.}}{5}} 10 \@writefile{lof}{\contentsline {figure}{\numberline {4}{\ignorespaces }}{4}} 11 \newlabel{fig:scheme}{{4}{4}} 12 \@writefile{lof}{\contentsline {figure}{\numberline {5}{\ignorespaces }}{5}} 13 \newlabel{fig:tag}{{5}{5}} 14 \@writefile{lof}{\contentsline {figure}{\numberline {6}{\ignorespaces This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1 and tag=65537. From the receiver's point of view, the endpoint 3 is actually receiving a message from MPI rank 0 with tag=65537.}}{5}} 15 \newlabel{fig:sendrecv}{{6}{5}} 16 \citation{ep:2018} 17 \@writefile{lof}{\contentsline {figure}{\numberline {7}{\ignorespaces }}{6}} 18 \newlabel{fig:bcast}{{7}{6}} 19 \@writefile{lof}{\contentsline {figure}{\numberline {8}{\ignorespaces }}{6}} 20 \newlabel{fig:allreduce}{{8}{6}} 21 \citation{ep:2018} 14 22 \bibstyle{plain} 15 23 \bibdata{reference} 16 \bibcite{Dinan:2013}{1} 17 \bibcite{Sridharan:2014}{2} 18 \@writefile{toc}{\contentsline {section}{\numberline {4}Performance of EP\_XIOS}{6}} 19 \@writefile{toc}{\contentsline {section}{\numberline {5}Perspectives of EP\_XIOS}{6}} 24 \bibcite{ep:2018}{1} 25 \bibcite{Dinan:2013}{2} 26 \bibcite{Sridharan:2014}{3} 27 \@writefile{toc}{\contentsline {section}{\numberline {3}The multi-threaded XIOS and performance results}{7}} 28 \@writefile{toc}{\contentsline {section}{\numberline {4}Future works for XIOS}{7}} -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.bbl
r1551 r1552 1 1 \begin{thebibliography}{1} 2 3 \bibitem{ep:2018} 4 XIOS developper group. 5 \newblock Note for {XIOS} {E}ndpoints. 6 \newblock Technical report, 7 \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp/Note}, 8 2018. 2 9 3 10 \bibitem{Dinan:2013} 4 11 J.~Dinan, Pavan Balaji, D.~Goodell, D.~Miller, M.~Snir, and Rajeev Thakur. 5 \newblock Enabling mpi interoperability through flexible communication6 endpoints.12 \newblock Enabling {MPI} {I}nteroperability {T}hrough {F}lexible 13 {C}ommunication {E}ndpoints. 7 14 \newblock In {\em EuroMPI 2013}, Madrid, Spain, 2013. 8 15 9 16 \bibitem{Sridharan:2014} 10 17 S.~Sridharan, J.~Dinan, and D.~D. Kalamkar. 11 \newblock Enabling efficient multithreaded mpi communication through a12 library-based implementation of mpi endpoints.18 \newblock Enabling {E}fficient {M}ultithreaded {MPI} {C}ommunication {T}hrough 19 a {L}ibrary-{B}ased {I}mplementation of {MPI} {E}ndpoints. 13 20 \newblock In {\em SC14: International Conference for High Performance 14 21 Computing, Networking, Storage and Analysis}, pages 487--498, Nov 2014. -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.blg
r1551 r1552 1 This is BibTeX, Version 0.99d (TeX Live 201 6)1 This is BibTeX, Version 0.99d (TeX Live 2015/Debian) 2 2 Capacity: max_strings=35307, hash_size=35307, hash_prime=30011 3 3 The top-level auxiliary file: rapport ESIWACE.aux 4 4 The style file: plain.bst 5 5 Database file #1: reference.bib 6 You've used 2 entries, 6 Warning--empty institution in ep:2018 7 You've used 3 entries, 7 8 2118 wiz_defined-function locations, 8 51 2 strings with 4502 characters,9 and the built_in function-call counts, 893in all, are:10 = -- 8011 > -- 5 19 517 strings with 4672 characters, 10 and the built_in function-call counts, 1094 in all, are: 11 = -- 101 12 > -- 58 12 13 < -- 1 13 + -- 2 014 - -- 1815 * -- 6316 := -- 1 4617 add.period$ -- 618 call.type$ -- 219 change.case$ -- 1514 + -- 23 15 - -- 20 16 * -- 73 17 := -- 186 18 add.period$ -- 9 19 call.type$ -- 3 20 change.case$ -- 20 20 21 chr.to.int$ -- 0 21 cite$ -- 222 duplicate$ -- 3723 empty$ -- 6824 format.name$ -- 1825 if$ -- 19222 cite$ -- 4 23 duplicate$ -- 45 24 empty$ -- 83 25 format.name$ -- 20 26 if$ -- 231 26 27 int.to.chr$ -- 0 27 int.to.str$ -- 228 int.to.str$ -- 3 28 29 missing$ -- 2 29 newline$ -- 1 330 num.names$ -- 431 pop$ -- 2 230 newline$ -- 18 31 num.names$ -- 6 32 pop$ -- 25 32 33 preamble$ -- 1 33 purify$ -- 1 334 purify$ -- 16 34 35 quote$ -- 0 35 skip$ -- 2536 skip$ -- 30 36 37 stack$ -- 0 37 substring$ -- 3838 substring$ -- 43 38 39 swap$ -- 12 39 40 text.length$ -- 1 40 41 text.prefix$ -- 0 41 42 top$ -- 0 42 type$ -- 8 43 warning$ -- 0 44 while$ -- 6 45 width$ -- 3 46 write$ -- 24 43 type$ -- 12 44 warning$ -- 1 45 while$ -- 8 46 width$ -- 4 47 write$ -- 35 48 (There was 1 warning) -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.log
r1551 r1552 1 This is pdfTeX, Version 3.14159265-2.6-1.40.1 7 (TeX Live 2016) (preloaded format=pdflatex 2016.7.25) 25 JUN 2018 17:201 This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015/Debian) (preloaded format=pdflatex 2017.8.24) 27 JUN 2018 15:10 2 2 entering extended mode 3 3 restricted \write18 enabled. … … 5 5 **"rapport ESIWACE.tex" 6 6 (./rapport ESIWACE.tex 7 LaTeX2e <2016/0 3/31>8 Babel <3.9 r> and hyphenation patterns for 83 language(s) loaded.9 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/article.cls7 LaTeX2e <2016/02/01> 8 Babel <3.9q> and hyphenation patterns for 3 language(s) loaded. 9 (/usr/share/texlive/texmf-dist/tex/latex/base/article.cls 10 10 Document Class: article 2014/09/29 v1.4h Standard LaTeX document class 11 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/size10.clo11 (/usr/share/texlive/texmf-dist/tex/latex/base/size10.clo 12 12 File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option) 13 13 ) … … 24 24 \bibindent=\dimen102 25 25 ) 26 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/inputenc.sty26 (/usr/share/texlive/texmf-dist/tex/latex/base/inputenc.sty 27 27 Package: inputenc 2015/03/17 v1.2c Input encoding file 28 28 \inpenc@prehook=\toks14 29 29 \inpenc@posthook=\toks15 30 30 31 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/utf8.def32 File: utf8.def 201 6/02/28 v1.1sUTF-8 support for inputenc31 (/usr/share/texlive/texmf-dist/tex/latex/base/utf8.def 32 File: utf8.def 2015/12/03 v1.1r UTF-8 support for inputenc 33 33 Now handling font encoding OML ... 34 34 ... no UTF-8 mapping file for font encoding OML … … 36 36 ... processing UTF-8 mapping file for font encoding T1 37 37 38 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/t1enc.dfu39 File: t1enc.dfu 201 6/02/28 v1.1sUTF-8 support for inputenc38 (/usr/share/texlive/texmf-dist/tex/latex/base/t1enc.dfu 39 File: t1enc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 40 40 defining Unicode char U+00A0 (decimal 160) 41 41 defining Unicode char U+00A1 (decimal 161) … … 249 249 defining Unicode char U+021A (decimal 538) 250 250 defining Unicode char U+021B (decimal 539) 251 defining Unicode char U+ 1E02 (decimal 7682)252 defining Unicode char U+ 1E03 (decimal 7683)251 defining Unicode char U+01E02 (decimal 7682) 252 defining Unicode char U+01E03 (decimal 7683) 253 253 defining Unicode char U+200C (decimal 8204) 254 defining Unicode char U+2010 (decimal 8208)255 defining Unicode char U+2011 (decimal 8209)256 defining Unicode char U+2012 (decimal 8210)257 254 defining Unicode char U+2013 (decimal 8211) 258 255 defining Unicode char U+2014 (decimal 8212) 259 defining Unicode char U+2015 (decimal 8213)260 256 defining Unicode char U+2018 (decimal 8216) 261 257 defining Unicode char U+2019 (decimal 8217) … … 273 269 ... processing UTF-8 mapping file for font encoding OT1 274 270 275 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/ot1enc.dfu276 File: ot1enc.dfu 201 6/02/28 v1.1sUTF-8 support for inputenc271 (/usr/share/texlive/texmf-dist/tex/latex/base/ot1enc.dfu 272 File: ot1enc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 277 273 defining Unicode char U+00A0 (decimal 160) 278 274 defining Unicode char U+00A1 (decimal 161) … … 314 310 ... processing UTF-8 mapping file for font encoding OMS 315 311 316 (/usr/ local/texlive/2016/texmf-dist/tex/latex/base/omsenc.dfu317 File: omsenc.dfu 201 6/02/28 v1.1sUTF-8 support for inputenc312 (/usr/share/texlive/texmf-dist/tex/latex/base/omsenc.dfu 313 File: omsenc.dfu 2015/12/03 v1.1r UTF-8 support for inputenc 318 314 defining Unicode char U+00A7 (decimal 167) 319 315 defining Unicode char U+00B6 (decimal 182) … … 338 334 defining Unicode char U+2423 (decimal 9251) 339 335 )) 340 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics/graphicx.sty336 (/usr/share/texlive/texmf-dist/tex/latex/graphics/graphicx.sty 341 337 Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR) 342 338 343 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics/keyval.sty339 (/usr/share/texlive/texmf-dist/tex/latex/graphics/keyval.sty 344 340 Package: keyval 2014/10/28 v1.15 key=value parser (DPC) 345 341 \KV@toks@=\toks16 346 342 ) 347 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics/graphics.sty348 Package: graphics 2016/0 5/09 v1.0rStandard LaTeX Graphics (DPC,SPQR)349 350 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics/trig.sty343 (/usr/share/texlive/texmf-dist/tex/latex/graphics/graphics.sty 344 Package: graphics 2016/01/03 v1.0q Standard LaTeX Graphics (DPC,SPQR) 345 346 (/usr/share/texlive/texmf-dist/tex/latex/graphics/trig.sty 351 347 Package: trig 2016/01/03 v1.10 sin cos tan (DPC) 352 348 ) 353 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics-cfg/graphics.cfg354 File: graphics.cfg 201 6/01/02 v1.10 sample graphics configuration355 ) 356 Package graphics Info: Driver file: pdftex.def on input line 9 6.357 358 (/usr/ local/texlive/2016/texmf-dist/tex/latex/pdftex-def/pdftex.def349 (/usr/share/texlive/texmf-dist/tex/latex/latexconfig/graphics.cfg 350 File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live 351 ) 352 Package graphics Info: Driver file: pdftex.def on input line 95. 353 354 (/usr/share/texlive/texmf-dist/tex/latex/pdftex-def/pdftex.def 359 355 File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX 360 356 361 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/infwarerr.sty362 Package: infwarerr 201 6/05/16 v1.4Providing info/warning/error messages (HO)363 ) 364 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/ltxcmds.sty365 Package: ltxcmds 201 6/05/16 v1.23LaTeX kernel commands for general use (HO)357 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/infwarerr.sty 358 Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO) 359 ) 360 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ltxcmds.sty 361 Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO) 366 362 ) 367 363 \Gread@gobject=\count87 … … 370 366 \Gin@req@width=\dimen104 371 367 ) 372 (/usr/ local/texlive/2016/texmf-dist/tex/latex/listings/listings.sty368 (/usr/share/texlive/texmf-dist/tex/latex/listings/listings.sty 373 369 \lst@mode=\count88 374 370 \lst@gtempboxa=\box26 … … 384 380 \lst@maxwidth=\dimen108 385 381 386 (/usr/ local/texlive/2016/texmf-dist/tex/latex/listings/lstmisc.sty382 (/usr/share/texlive/texmf-dist/tex/latex/listings/lstmisc.sty 387 383 File: lstmisc.sty 2015/06/04 1.6 (Carsten Heinz) 388 384 \c@lstnumber=\count94 … … 390 386 \lst@framebox=\box27 391 387 ) 392 (/usr/ local/texlive/2016/texmf-dist/tex/latex/listings/listings.cfg388 (/usr/share/texlive/texmf-dist/tex/latex/listings/listings.cfg 393 389 File: listings.cfg 2015/06/04 1.6 listings configuration 394 390 )) 395 391 Package: listings 2015/06/04 1.6 (Carsten Heinz) 396 392 397 (/usr/ local/texlive/2016/texmf-dist/tex/latex/xcolor/xcolor.sty398 Package: xcolor 20 16/05/11 v2.12LaTeX color extensions (UK)399 400 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics-cfg/color.cfg401 File: color.cfg 20 16/01/02 v1.6 sample color configuration393 (/usr/share/texlive/texmf-dist/tex/latex/xcolor/xcolor.sty 394 Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK) 395 396 (/usr/share/texlive/texmf-dist/tex/latex/latexconfig/color.cfg 397 File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive 402 398 ) 403 399 Package xcolor Info: Package option `usenames' ignored on input line 216. 404 400 Package xcolor Info: Driver file: pdftex.def on input line 225. 405 401 406 (/usr/ local/texlive/2016/texmf-dist/tex/latex/colortbl/colortbl.sty402 (/usr/share/texlive/texmf-dist/tex/latex/colortbl/colortbl.sty 407 403 Package: colortbl 2012/02/13 v1.0a Color table columns (DPC) 408 404 409 (/usr/ local/texlive/2016/texmf-dist/tex/latex/tools/array.sty405 (/usr/share/texlive/texmf-dist/tex/latex/tools/array.sty 410 406 Package: array 2014/10/28 v2.4c Tabular extension package (FMi) 411 407 \col@sep=\dimen109 … … 419 415 ) 420 416 \rownum=\count96 421 Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 13 48.422 Package xcolor Info: Model `hsb' substituted by `rgb' on input line 13 52.423 Package xcolor Info: Model `RGB' extended on input line 13 64.424 Package xcolor Info: Model `HTML' substituted by `rgb' on input line 13 66.425 Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 13 67.426 Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 13 68.427 Package xcolor Info: Model `HSB' substituted by `hsb' on input line 13 69.428 Package xcolor Info: Model `Gray' substituted by `gray' on input line 13 70.429 Package xcolor Info: Model `wave' substituted by `hsb' on input line 13 71.430 431 (/usr/ local/texlive/2016/texmf-dist/tex/latex/graphics/dvipsnam.def417 Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1337. 418 Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1341. 419 Package xcolor Info: Model `RGB' extended on input line 1353. 420 Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1355. 421 Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1356. 422 Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1357. 423 Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1358. 424 Package xcolor Info: Model `Gray' substituted by `gray' on input line 1359. 425 Package xcolor Info: Model `wave' substituted by `hsb' on input line 1360. 426 427 (/usr/share/texlive/texmf-dist/tex/latex/graphics/dvipsnam.def 432 428 File: dvipsnam.def 2015/12/30 v3.0k Driver-dependent file (DPC,SPQR) 433 429 ) 434 (/usr/ local/texlive/2016/texmf-dist/tex/latex/xcolor/svgnam.def435 File: svgnam.def 20 16/05/11 v2.12Predefined colors according to SVG 1.1 (UK)430 (/usr/share/texlive/texmf-dist/tex/latex/xcolor/svgnam.def 431 File: svgnam.def 2007/01/21 v2.11 Predefined colors according to SVG 1.1 (UK) 436 432 )) 437 (/usr/ local/texlive/2016/texmf-dist/tex/latex/amsmath/amsmath.sty438 Package: amsmath 2016/03/ 10 v2.15bAMS math features433 (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsmath.sty 434 Package: amsmath 2016/03/03 v2.15a AMS math features 439 435 \@mathmargin=\skip46 440 436 441 437 For additional information on amsmath, use the `?' option. 442 (/usr/ local/texlive/2016/texmf-dist/tex/latex/amsmath/amstext.sty438 (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amstext.sty 443 439 Package: amstext 2000/06/29 v2.01 AMS text 444 440 445 (/usr/ local/texlive/2016/texmf-dist/tex/latex/amsmath/amsgen.sty441 (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsgen.sty 446 442 File: amsgen.sty 1999/11/30 v2.0 generic functions 447 443 \@emptytoks=\toks20 448 444 \ex@=\dimen111 449 445 )) 450 (/usr/ local/texlive/2016/texmf-dist/tex/latex/amsmath/amsbsy.sty446 (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsbsy.sty 451 447 Package: amsbsy 1999/11/29 v1.2d Bold Symbols 452 448 \pmbraise@=\dimen112 453 449 ) 454 (/usr/ local/texlive/2016/texmf-dist/tex/latex/amsmath/amsopn.sty455 Package: amsopn 2016/03/08 v2.02operator names450 (/usr/share/texlive/texmf-dist/tex/latex/amsmath/amsopn.sty 451 Package: amsopn 1999/12/14 v2.01 operator names 456 452 ) 457 453 \inf@bad=\count97 … … 468 464 \strutbox@=\box29 469 465 \big@size=\dimen113 470 LaTeX Font Info: Redeclaring font encoding OML on input line 63 4.471 LaTeX Font Info: Redeclaring font encoding OMS on input line 63 5.466 LaTeX Font Info: Redeclaring font encoding OML on input line 630. 467 LaTeX Font Info: Redeclaring font encoding OMS on input line 631. 472 468 \macc@depth=\count102 473 469 \c@MaxMatrixCols=\count103 … … 490 486 \multlinetaggap=\skip48 491 487 \mathdisplay@stack=\toks24 492 LaTeX Info: Redefining \[ on input line 273 9.493 LaTeX Info: Redefining \] on input line 27 40.494 ) 495 (/usr/ local/texlive/2016/texmf-dist/tex/latex/url/url.sty488 LaTeX Info: Redefining \[ on input line 2735. 489 LaTeX Info: Redefining \] on input line 2736. 490 ) 491 (/usr/share/texlive/texmf-dist/tex/latex/url/url.sty 496 492 \Urlmuskip=\muskip11 497 493 Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc. 498 494 ) 495 (/usr/share/texlive/texmf-dist/tex/latex/tools/verbatim.sty 496 Package: verbatim 2014/10/28 v1.5q LaTeX2e package for verbatim enhancements 497 \every@verbatim=\toks25 498 \verbatim@line=\toks26 499 \verbatim@in@stream=\read1 500 ) 499 501 (./rapport ESIWACE.aux) 500 502 \openout1 = `"rapport ESIWACE.aux"'. 501 503 502 LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 1 6.503 LaTeX Font Info: ... okay on input line 1 6.504 LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 1 6.505 LaTeX Font Info: ... okay on input line 1 6.506 LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 1 6.507 LaTeX Font Info: ... okay on input line 1 6.508 LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 1 6.509 LaTeX Font Info: ... okay on input line 1 6.510 LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 1 6.511 LaTeX Font Info: ... okay on input line 1 6.512 LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 1 6.513 LaTeX Font Info: ... okay on input line 1 6.514 515 (/usr/ local/texlive/2016/texmf-dist/tex/context/base/mkii/supp-pdf.mkii504 LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 17. 505 LaTeX Font Info: ... okay on input line 17. 506 LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 17. 507 LaTeX Font Info: ... okay on input line 17. 508 LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 17. 509 LaTeX Font Info: ... okay on input line 17. 510 LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 17. 511 LaTeX Font Info: ... okay on input line 17. 512 LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 17. 513 LaTeX Font Info: ... okay on input line 17. 514 LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 17. 515 LaTeX Font Info: ... okay on input line 17. 516 517 (/usr/share/texlive/texmf-dist/tex/context/base/supp-pdf.mkii 516 518 [Loading MPS to PDF converter (version 2006.09.02).] 517 519 \scratchcounter=\count109 … … 520 522 \nofMPsegments=\count110 521 523 \nofMParguments=\count111 522 \everyMPshowfont=\toks2 5524 \everyMPshowfont=\toks27 523 525 \MPscratchCnt=\count112 524 526 \MPscratchDim=\dimen121 525 527 \MPnumerator=\count113 526 528 \makeMPintoPDFobject=\count114 527 \everyMPtoPDFconversion=\toks2 6528 ) (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty529 Package: pdftexcmds 201 6/05/10 v0.21Utility functions of pdfTeX for LuaTeX (HO530 ) 531 532 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/ifluatex.sty533 Package: ifluatex 201 6/05/16 v1.4Provides the ifluatex switch (HO)529 \everyMPtoPDFconversion=\toks28 530 ) (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty 531 Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO 532 ) 533 534 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifluatex.sty 535 Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO) 534 536 Package ifluatex Info: LuaTeX not detected. 535 537 ) 536 (/usr/local/texlive/2016/texmf-dist/tex/generic/oberdiek/ifpdf.sty 537 Package: ifpdf 2016/05/14 v3.1 Provides the ifpdf switch 538 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/ifpdf.sty 539 Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO) 540 Package ifpdf Info: pdfTeX in PDF mode is detected. 538 541 ) 539 542 Package pdftexcmds Info: LuaTeX not detected. … … 542 545 Package pdftexcmds Info: \pdfdraftmode found. 543 546 ) 544 (/usr/ local/texlive/2016/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty545 Package: epstopdf-base 201 6/05/15 v2.6Base part for package epstopdf546 547 (/usr/ local/texlive/2016/texmf-dist/tex/latex/oberdiek/grfext.sty548 Package: grfext 201 6/05/16 v1.2Manage graphics extensions (HO)549 550 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty551 Package: kvdefinekeys 201 6/05/16 v1.4Define keys (HO)547 (/usr/share/texlive/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty 548 Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf 549 550 (/usr/share/texlive/texmf-dist/tex/latex/oberdiek/grfext.sty 551 Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO) 552 553 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/kvdefinekeys.sty 554 Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO) 552 555 )) 553 (/usr/ local/texlive/2016/texmf-dist/tex/latex/oberdiek/kvoptions.sty554 Package: kvoptions 201 6/05/16 v3.12Key value format for package options (HO)555 556 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty557 Package: kvsetkeys 201 6/05/16 v1.17Key value parser (HO)558 559 (/usr/ local/texlive/2016/texmf-dist/tex/generic/oberdiek/etexcmds.sty560 Package: etexcmds 201 6/05/16 v1.6Avoid name clashes with e-TeX commands (HO)556 (/usr/share/texlive/texmf-dist/tex/latex/oberdiek/kvoptions.sty 557 Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO) 558 559 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty 560 Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO) 561 562 (/usr/share/texlive/texmf-dist/tex/generic/oberdiek/etexcmds.sty 563 Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO) 561 564 Package etexcmds Info: Could not find \expanded. 562 565 (etexcmds) That can mean that you are not using pdfTeX 1.50 or … … 567 570 (grfext) [.png,.pdf,.jpg,.mps,.jpeg,.jbig2,.jb2,.PNG,.PDF,.JPG,.JPE 568 571 G,.JBIG2,.JB2,.eps] 569 (grfext) \AppendGraphicsExtensions on input line 45 6.570 571 (/usr/ local/texlive/2016/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg572 (grfext) \AppendGraphicsExtensions on input line 452. 573 574 (/usr/share/texlive/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg 572 575 File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv 573 576 e … … 578 581 File: Charge1.png Graphic file (type png) 579 582 <use Charge1.png> 580 Package pdftex.def Info: Charge1.png used on input line 3 2.583 Package pdftex.def Info: Charge1.png used on input line 33. 581 584 (pdftex.def) Requested size: 165.01357pt x 91.23924pt. 582 585 … … 584 587 File: Charge2.png Graphic file (type png) 585 588 <use Charge2.png> 586 Package pdftex.def Info: Charge2.png used on input line 3 3.589 Package pdftex.def Info: Charge2.png used on input line 34. 587 590 (pdftex.def) Requested size: 165.25446pt x 91.05858pt. 588 591 [1 589 592 590 {/usr/local/texlive/2016/texmf-var/fonts/map/pdftex/updmap/pdftex.map} <./Charg 591 e1.png> <./Charge2.png>] <domain.pdf, id=16, 472.31454pt x 142.26149pt> 593 {/var/lib/texmf/fonts/map/pdftex/updmap/pdftex.map} <./Charge1.png> <./Charge2. 594 png>] 595 <domain.pdf, id=16, 472.31454pt x 142.26149pt> 592 596 File: domain.pdf Graphic file (type pdf) 593 597 <use domain.pdf> 594 Package pdftex.def Info: domain.pdf used on input line 59.598 Package pdftex.def Info: domain.pdf used on input line 60. 595 599 (pdftex.def) Requested size: 236.1567pt x 71.13055pt. 596 600 … … 598 602 File: omp.pdf Graphic file (type pdf) 599 603 <use omp.pdf> 600 Package pdftex.def Info: omp.pdf used on input line 6 7.604 Package pdftex.def Info: omp.pdf used on input line 68. 601 605 (pdftex.def) Requested size: 291.64784pt x 126.32893pt. 602 603 604 LaTeX Warning: `!h' float specifier changed to `!ht'. 605 606 [2 <./domain.pdf>] <scheme.png, id=27, 665.48625pt x 517.935pt> 606 [2 <./domain.pdf>] 607 <scheme.png, id=27, 665.48625pt x 517.935pt> 607 608 File: scheme.png Graphic file (type png) 608 609 <use scheme.png> 610 Package pdftex.def Info: scheme.png used on input line 79. 609 <use scheme.png> 610 Package pdftex.def Info: scheme.png used on input line 86. 611 611 (pdftex.def) Requested size: 266.18977pt x 207.17032pt. 612 [3 <./omp.pdf> <./scheme.png (PNG copy)>] [4] 613 <LMDZ_perf.png, id=39, 530.73282pt x 255.95625pt> 614 File: LMDZ_perf.png Graphic file (type png) 615 <use LMDZ_perf.png> 616 Package pdftex.def Info: LMDZ_perf.png used on input line 144. 617 (pdftex.def) Requested size: 318.44214pt x 153.57492pt. 618 [5 <./LMDZ_perf.png (PNG copy)>] (./rapport ESIWACE.bbl) [6] (./rapport ESIWAC 619 E.aux) ) 612 [3 <./omp.pdf>] 613 <tag.png, id=36, 752.8125pt x 103.38625pt> 614 File: tag.png Graphic file (type png) 615 <use tag.png> 616 Package pdftex.def Info: tag.png used on input line 140. 617 (pdftex.def) Requested size: 301.11966pt x 41.35376pt. 618 [4 <./scheme.png (PNG copy)>] <sendrecv.png, id=41, 829.0975pt x 694.595pt> 619 File: sendrecv.png Graphic file (type png) 620 <use sendrecv.png> 621 Package pdftex.def Info: sendrecv.png used on input line 149. 622 (pdftex.def) Requested size: 331.63313pt x 277.83307pt. 623 624 [5 <./tag.png (PNG copy)> <./sendrecv.png (PNG copy)>] 625 <bcast.png, id=45, 512.91624pt x 552.0625pt> 626 File: bcast.png Graphic file (type png) 627 <use bcast.png> 628 Package pdftex.def Info: bcast.png used on input line 173. 629 (pdftex.def) Requested size: 153.87605pt x 165.62003pt. 630 631 <allreduce.png, id=46, 743.77875pt x 552.0625pt> 632 File: allreduce.png Graphic file (type png) 633 <use allreduce.png> 634 Package pdftex.def Info: allreduce.png used on input line 185. 635 (pdftex.def) Requested size: 223.13535pt x 165.62003pt. 636 [6 <./bcast.png (PNG copy)> <./allreduce.png (PNG copy)>] (./rapport ESIWACE.b 637 bl 638 Underfull \hbox (badness 3354) in paragraph at lines 4--9 639 []\OT1/cmr/m/n/10 XIOS de-vel-op-per group. Note for XIOS End-points. Tech-ni 640 -cal 641 [] 642 643 644 Underfull \hbox (badness 4819) in paragraph at lines 4--9 645 \OT1/cmr/m/n/10 re-port, $\OT1/cmtt/m/n/10 http : / / forge . ipsl . jussieu . 646 fr / ioserver / browser / XIOS / dev / 647 [] 648 649 ) [7] (./rapport ESIWACE.aux) ) 620 650 Here is how much of TeX's memory you used: 621 4 595 strings out of 493014622 61 054 string characters out of 6133351623 1 42700words of memory out of 5000000624 8068multiletter control sequences out of 15000+600000651 4637 strings out of 494953 652 61251 string characters out of 6180977 653 137489 words of memory out of 5000000 654 7849 multiletter control sequences out of 15000+600000 625 655 9090 words of font info for 34 fonts, out of 8000000 for 9000 626 1 141hyphenation exceptions out of 8191656 14 hyphenation exceptions out of 8191 627 657 41i,8n,35p,1270b,264s stack positions out of 5000i,500n,10000p,200000b,80000s 628 </usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmbx 629 12.pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi1 630 0.pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi7. 631 pfb></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pf 632 b></usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr12.pfb> 633 </usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr17.pfb></ 634 usr/local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr6.pfb></usr 635 /local/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr7.pfb></usr/lo 636 cal/texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmr8.pfb></usr/local 637 /texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb></usr/local/ 638 texlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmti10.pfb></usr/local/t 639 exlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/local/te 640 xlive/2016/texmf-dist/fonts/type1/public/amsfonts/cm/cmtt8.pfb> 641 Output written on "rapport ESIWACE.pdf" (6 pages, 229957 bytes). 658 </usr/share/texlive/texmf-dist/fonts/type1/publi 659 c/amsfonts/cm/cmbx12.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsf 660 onts/cm/cmr10.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm 661 /cmr12.pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr17. 662 pfb></usr/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr6.pfb></us 663 r/share/texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr7.pfb></usr/share/ 664 texlive/texmf-dist/fonts/type1/public/amsfonts/cm/cmr8.pfb></usr/share/texlive/ 665 texmf-dist/fonts/type1/public/amsfonts/cm/cmti10.pfb></usr/share/texlive/texmf- 666 dist/fonts/type1/public/amsfonts/cm/cmtt10.pfb></usr/share/texlive/texmf-dist/f 667 onts/type1/public/amsfonts/cm/cmtt8.pfb> 668 Output written on "rapport ESIWACE.pdf" (7 pages, 269577 bytes). 642 669 PDF statistics: 643 90PDF objects out of 1000 (max. 8388607)644 62compressed objects within 1 object stream670 86 PDF objects out of 1000 (max. 8388607) 671 57 compressed objects within 1 object stream 645 672 0 named destinations out of 1000 (max. 500000) 646 31words of extra memory for PDF output out of 10000 (max. 10000000)647 673 46 words of extra memory for PDF output out of 10000 (max. 10000000) 674 -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.tex
r1551 r1552 6 6 \usepackage{amsmath} 7 7 \usepackage{url} 8 \usepackage{verbatim} 8 9 9 10 % Title Page … … 29 30 writing on the server side can be overlapped completely by calculates on the client side. 30 31 31 \begin{figure}[h ]32 \begin{figure}[ht] 32 33 \includegraphics[scale=0.4]{Charge1.png} 33 34 \includegraphics[scale=0.4]{Charge2.png} … … 55 56 This topic comes along with the configuration of the climate models. Take LMDZ as example, it is designed with the 2-level parallelization scheme. To be more specific, LMDZ uses the domain decomposition method in which each sub-domain is associated with one MPI process. Inside of the sub-domain, the model also uses OpenMP derivatives to accelerate the computation. We can imagine that the sub-domain be divided into sub-sub-domain and is managed by threads. 56 57 57 \begin{figure}[h ]58 \begin{figure}[ht] 58 59 \centering 59 60 \includegraphics[scale=0.5]{domain.pdf} … … 63 64 As we know, each sub-domain, or in another word, each MPI process is a XIOS client. The data exchange between client and XIOS servers is handled by MPI communications. In order to write an output field, all threads must gather the data to the master thread who acts as MPI process in order to call MPI routines. There are two disadvantages about this method : first, we have to spend time on gathering information to the master thread which not only increases the memory use, but also implies an OpenMP barrier; second, while the master thread calls MPI routine, other threads are in the idle state thus a waster of computing resources. What we want obtain with the thread-friendly XIOS is that all threads can act like MPI processes. They can call directly the MPI routine thus no waste in memory nor in computing resources as shown in Figure \ref{fig:omp}. 64 65 65 \begin{figure}[h !]66 \begin{figure}[ht] 66 67 \centering 67 68 \includegraphics[scale=0.6]{omp.pdf} … … 73 74 74 75 75 In this project, we choose to implement the interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a concept proposed in the last MPI Forum and several papers has already discussed the importance of such idea and have introduced the framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. Threads of an MPI process is associated with a unique rank (global endpoint rank) and an endpoint communicator. They also have a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num| rank. 76 77 \begin{figure}[h!] 76 In this project, we choose to implement an interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a 77 concept proposed in the last MPI Forums and several papers have already discussed the importance of such idea and have introduced the 78 framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. In 79 the MPI\_endpoint environment, each OpenMP thread will be associated with a unique rank (global endpoint rank), an endpoint communicator, 80 and a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num|. The global endpoint rank will replace the 81 role of the classic MPI rank and will be used in MPI communication calls. 82 83 84 \begin{figure}[ht] 78 85 \begin{center} 79 86 \includegraphics[scale=0.4]{scheme.png} … … 85 92 %XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we introduce the MPI endpoints. 86 93 87 88 The MPI\_endpoints interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions used in XIOS. 89 90 will be re-implemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be 91 associated with a unique rank and with an endpoint communicator. This rank (EP rank) will replace the role of the classic MPI rank and will 92 be used in MPI communications. In order to successfully execute an MPI communication, for example \verb|MPI_Send|, we know already which 93 endpoints to be the receiver but not sufficient. We also need to know which MPI process should be involved in such communication. To 94 identify the MPI rank, we added a ``map'' in the EP communicator in which the relation of all EP and MPI ranks can be easily obtained. 95 96 97 In XIOS, we used the ``probe'' technique to search for arrived messages and then performing the receive action. The principle is 98 that sender processes execute the send operations as usual. However, to minimize the time spent on waiting incoming messages, the receiver 99 processe performs in the first place the \verb|MPI_Probe| function to check if a message destinated to it has been published. If yes, the 100 process execute in the second place the \verb|MPI_Recv| to receive the message. In this situation, if we introduce the threads, problems 101 occur. The reason why the ``probe'' method is not suitable is that messages destinated to one certain process can be probed by any of 102 its threads. Thus the message can be received by the wrong thread which gives errors. 103 104 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each process is equiped with a local 105 incoming message queue. All incoming message will be probed, sorted, and then stored in this queue according to their destination rank. 106 Every time we call an MPI function, we firstly call the \verb|MPI_Mprobe| function to get the handle to 107 the incoming message. Then, we identify the destination thread rank and store the message handle inside the local queue of the target 108 thread. After this, we perform the usual ``probe'' technique upon the local incoming message queue. In this way, we can assure the messages 109 to be received by the right thread. 110 111 Another issue remains in this technique: how to identify the receiver's rank? The solution is to use the tag argument. In the MPI 112 environment, a tag is an integer ranging from 0 to $2^{31}$. We can explore the large range of the tag to store in it information about the 113 source and destination thread ranks. We choose to limite the first 15 bits for the tag used in the classic MPI communication, the next 8 114 bits to the sender's thread rank, and the last 8 bits to the receiver's thread rank. In such way, with an extra analysis of the EP tag, we 115 can identify the ranks of the sender and the receiver in any P2P communication. As results, we a thread probes a message, it knows 116 exactly in which local queue should store the probed message. 117 118 119 With the global rank map, tag extension, and the matching-probe techniques, we are able to use any P2P communication in the endpoint 120 environment. For the collective communications, we perform a step-by-step execution and no special technique is required. The most 121 representative functions is the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. A step-by-step execution consists of 122 3 steps (not necessarily in this order): arrangement of the source data, execution of the MPI function by all 123 master/root threads, distribution or arrangement of the data among threads. 124 125 For example, if we want to perform a broadcast operation, 2 steps are needed. Firstly, the root thread, along with the master threads of 126 other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the master threads send data to threads 127 sharing the same process via local memory transfer. In another example for illustrating the \verb|MPI_Gather| function, we also need 2 128 steps. First of all, data is gathered from slave threads to the master thread or the root thread. Next, the master thread and the root 129 thread execute the \verb|MPI_Gather| operation of complete the communication. Other collective calls such as \verb|MPI_Scan|, 130 \verb|MPI_Reduce|, \verb|MPI_Scatter| \textit{etc} follow the same principle of step-by-step execution. 131 132 94 An other important aspect about the MPI\_endpoint interface is that each endpoints has knowledge of the ranks of other endpoints in the 95 same communicator. This knowledge is necessary because when executing an MPI communication, for example a point-to-point exchange, we need 96 to know not only the ranks of sender/receiver threads, but also the thread number of the sender/receiver threads and the MPI ranks 97 of the sender/receiver processes. This ranking information is implemented inside an map object included in the endpoint communicator 98 class. 99 100 101 102 103 %\newpage 104 105 %The MPI\_endpoint interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions 106 %used in XIOS. 107 108 109 In XIOS, we used the ``probe'' technique to search for arrived messages and then perform the receiving action. The principle is 110 that sender process executes the send operation as usual. However, to minimize the time spent on waiting incoming messages, the receiver 111 process calls in the first place the \verb|MPI_Probe| function to check if a message destinate to it has been published. If yes, the 112 process execute in the second place the \verb|MPI_Recv| to receive the message. If not, the receiver process can carry on with other tasks 113 or repeats the \verb|MPI_Probe| and \verb|MPI_Recv| actions if the required message is in immediate need. This technique works well in 114 the current version of XIOS. However, if we introduce threads into this mechanism, problems can occur: The incoming message is labeled by 115 the tag and receiver's MPI rank. Because threads within a process share the MPI rank, and the message probed is always available in the 116 message queue, it can lead to the problem of data race and thus the message can be received by the wrong thread. 117 118 119 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each thread is equipped with a local 120 incoming message queue. Each time a thread calls an MPI function, for example \verb|MPI_Recv|, it calls firstly the \verb|MPI_Mprobe| 121 function to query the MPI incoming message with any tag and from any source. Once a message is probed, the thread gets the handle to the 122 incoming message and this specific message is erased from the MPI message queue. Then, the thread proceed the identification of the message 123 to get the destination thread's local rank and store the message handle to the local queue of the target thread. The thread repeats these 124 steps until the MPI incoming message queue is empty. Then the thread we perform the usual ``probe'' technique to query its local incoming 125 message queue to check if the required message is available. If yes, it performs the \verb|MPI_Recv| operation. With this ``matching-probe'' 126 technique, we can assure that a message is probed only once and is received by the right receiver. 127 128 129 130 Another issue needs to be clarified with this technique is that: how to identify the receiver's rank? The solution to this question is to 131 use the tag argument. In the MPI environment, a tag is an integer ranging from 0 to $2^{31}$ depending on the Implementation. We can explore 132 the large range property of the tag to store in it information about the source and destination thread ranks. In our endpoint interface, we 133 choose to limit the first 15 bits for the tag used in the classic MPI communication, the next 8 bits to store the sender thread's local 134 rank, and the last 8 bits to store the receiver thread's local rank (\textit{c.f.} Figure \ref{fig:tag}). In this way, with an extra 135 analysis of the tag, we can identify the local ranks of the sender and the receiver in any P2P communication. As results, when a thread 136 probes a message, it knows exactly in which local queue should store the probed message. 137 138 \begin{figure}[ht] 139 \centering 140 \includegraphics[scale=0.4]{tag.png} 141 \caption{}\label{fig:tag} 142 \end{figure} 143 144 In Figure \ref{fig:tag}, Tag contains the user defined value for a certain communication. MPI\_tag is computed in the endpoint interface 145 with help of the rank map and is used in the MPI calls. 146 147 \begin{figure}[ht] 148 \centering 149 \includegraphics[scale = 0.4]{sendrecv.png} 150 \caption{This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message 151 to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1 152 and tag=65537. From the receiver's point of view, the endpoint 3 is actually receiving a message from MPI rank 0 with 153 tag=65537.} 154 \label{fig:sendrecv} 155 \end{figure} 156 157 158 159 160 With the rank map, tag extension, and the matching-probe techniques, we are now able to call any P2P communication in the endpoint 161 environment. For the collective communications, we apply a step-by-step execution pattern and no special technique is required. A 162 step-by-step execution pattern consists of 3 steps (not necessarily in this order and not all steps are needed): arrangement of the source 163 data, execution of the MPI function by all master/root threads, distribution or arrangement of the resulting data among threads. 164 165 %The most representative functions of the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. 166 167 For example, if we want to perform a broadcast operation, only 2 steps are needed (\textit{c.f.} Figure \ref{fig:bcast}). Firstly, the root 168 thread, along with the master threads of other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the 169 master threads send data to other threads via local memory transfer. 170 171 \begin{figure}[ht] 172 \centering 173 \includegraphics[scale=0.3]{bcast.png} 174 \caption{} 175 \label{fig:bcast} 176 \end{figure} 177 178 Figure \ref{fig:allreduce} illustrates how the \verb|MPI_Allreduce| function is proceeded in the endpoint interface. First of all, We 179 perform a intra-process ``allreduce'' operation: source data is reduced from slave threads to the master thread via local memory transfer. 180 Next, all master threads call the classic \verb|MPI_Allreduce| routine. Finally, all master threads send the updated reduced data to its 181 slaves via local memory transfer. 182 183 \begin{figure}[ht] 184 \centering 185 \includegraphics[scale=0.3]{allreduce.png} 186 \caption{} 187 \label{fig:allreduce} 188 \end{figure} 189 190 Other MPI routines, such as \verb|MPI_Wait|, \verb|MPI_Intercomm_create| \textit{etc.}, can be found in the technique report of the 191 endpoint interface. 192 193 \section{The multi-threaded XIOS and performance results} 194 195 The development of endpoint interface for thread-friendly XIOS library took about one year and a half. The main difficulty is the 196 co-existence of MPI processes and OpenMP threads. All MPI classes must be redefined in the endpoint interface along with all the routines. 197 The development is now available on the forge server: \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp}. One 198 technique report is also available in which one can find more detail about how endpoint works and how the routines are implemented 199 \cite{ep:2018}. We must note that the thread-friendly XIOS library is still in the phase of optimization. It will be released in the 200 future with a stable version. 201 202 All the functionalities of XIOS is reserved in its thread-friendly version. Single threaded code can work successfully with the new 203 version of XIOS. For multi-threaded models, some modifications are needed in order to work with the multi-threaded XIOS library. Detail can 204 be found in our technique report \cite{ep:2018}. 205 206 Even though the multi-threaded XIOS library is not fully accomplished and further optimization in ongoing. We have already done some tests 207 to see the potential of the endpoint framework. We take LMDZ as the target model and have tested with several work-flow charges. 208 209 \begin{comment} 133 210 \section{Performance of LMDZ using EP\_XIOS} 134 211 … … 141 218 simulation duration settings: 1 day, 5 days, 15 days, and 31 days. 142 219 143 \begin{figure}[h ]220 \begin{figure}[ht] 144 221 \centering 145 222 \includegraphics[scale = 0.6]{LMDZ_perf.png} … … 167 244 histmth with daily output 168 245 169 \section{Perspectives of EP\_XIOS} 246 \end{comment} 247 248 249 \section{Future works for XIOS} 170 250 171 251 -
XIOS/dev/branch_openmp/Note/rapport ESIWACE.tex.backup
r1548 r1552 5 5 \usepackage[usenames,dvipsnames,svgnames,table]{xcolor} 6 6 \usepackage{amsmath} 7 \usepackage{url} 7 8 8 9 % Title Page 9 10 10 \title{Develop ping XIOS with multithread : to accelerate the IO of climate models}11 \title{Developing XIOS with multi-thread : to accelerate the I/O of climate models} 11 12 12 13 \author{} … … 16 17 \maketitle 17 18 18 \section{ background}19 \section{Context} 19 20 20 21 The simulation models of climate systems, running on a large number of computing resources can produce an important volume of data. At this 21 scale, the I O and the post-treatement of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux22 generated by the simulations, we use XIOS develop ped by the Institut Pierre Simon Laplace and Maison de la simulation.23 24 XIOS, a lib arary dedicated to intense calculates, allows us to easily and efficiently manage the parallel IO on the storage systems. XIOS22 scale, the I/O and the post-treatment of data becomes a bottle-neck for the performance. In order to manage efficiently the data flux 23 generated by the simulations, we use XIOS developed by the Institut Pierre Simon Laplace and Maison de la simulation. 24 25 XIOS, a library dedicated to intense calculates, allows us to easily and efficiently manage the parallel I/O on the storage systems. XIOS 25 26 uses the client/server scheme in which computing resources (server) are reserved exclusively for IO in order to minimize their impact on 26 the performance of the climate models (client). 27 28 Cette bibliothÚque, dédiée au calcul intensif, permet de gérer efficacement et simplement les 29 entrée/sortie parallÚles des données sur les systÚmes de stockage. Dans cette nouvelle 30 approche, orientée client/serveur, des cÅurs de calcul sont exclusivement dédiés aux I/O de 31 façon à minimiser leur impact sur le temps de calcul des modÚles. Lâutilisation des 32 communications asynchrones entre les modÚles (clients) et les serveurs I/O permet de lisser 33 les pics I/O en envoyant un flux de données constant au systÚme de fichiers tout au long de la 34 simulation, recouvrant ainsi totalement les écritures par du calcul. 35 36 37 The aim of this project ESIWACE is to develop a multithreaded version of XIOS, a library dedicated to IO manegement of climate code. 38 The current XIOS code lies on a single level of parallelization using MPI. However, many climate models are now disigned with two-level 39 parallelization through MPI and OpenMP. The difference of parallelization between the climate models and XIOS can lead to performance lost 40 because XIOS can not cope with threads. This fact 41 42 43 The resulting multithreaded XIOS is desinged to cope with climate models which use a two-level parallelization (MPI/Openmp) scheme. 44 The principle model we work with is the LMDZ code developped at Laboratoire de Météorologie Dynamique. This model has 45 46 47 48 \section{Developpement of a thread-friendly MPI for XIOS} 49 50 XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations 51 and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. 52 However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP 53 directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to 54 maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI 55 process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the 56 updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. 57 This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a 58 new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we 59 introduce the MPI endpoints. 60 61 62 The MPI endpoints (EP) is a layer on top of an existing MPI Implementation. All MPI function, or in our work the functions used in XIOS, 63 will be reimplemented in order to cope with OpenMP threads. The idea is that, in the MPI endpoints environment, each OpenMP thread will be 64 associated with a unique rank and with an endpoint communicator. This rank (EP rank) will replace the role of the classic MPI rank and will 65 be used in MPI communications. In order to successfully execute an MPI communication, for example \verb|MPI_Send|, we know already which 66 endpoints to be the receiver but not sufficient. We also need to know which MPI process should be involved in such communication. To 67 identify the MPI rank, we added a ``map'' in the EP communicator in which the relation of all EP and MPI ranks can be easily obtained. 68 69 70 In XIOS, we used the ``probe'' technique to search for arrived messages and then performing the receive action. The principle is 71 that sender processes execute the send operations as usual. However, to minimise the time spent on waiting incoming messages, the receiver 72 processe performs in the first place the \verb|MPI_Probe| function to check if a message destinated to it has been published. If yes, the 73 process execute in the second place the \verb|MPI_Recv| to receive the message. In this situation, if we introduce the threads, problems 74 occur. The reason why the ``probe'' method is not suitable is that messages destinated to one certain process can be probed by any of 75 its threads. Thus the message can be received by the wrong thread which gives errors. 76 77 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each process is equiped with a local 78 incoming message queue. All incoming message will be probed, sorted, and then stored in this queue according to their destination rank. 79 Every time we call an MPI function, we firstly call the \verb|MPI_Mprobe| function to get the handle to 80 the incoming message. Then, we identify the destination thread rank and store the message handle inside the local queue of the target 81 thread. After this, we perform the usual ``probe'' technique upon the local incoming message queue. In this way, we can assure the messages 82 to be received by the right thread. 83 84 Another issue remains in this technique: how to identify the receiver's rank? The solution is to use the tag argument. In the MPI 85 environment, a tag is an integer ranging from 0 to $2^{31}$. We can explore the large range of the tag to store in it information about the 86 source and destination thread ranks. We choose to limite the first 15 bits for the tag used in the classic MPI communication, the next 8 87 bits to the sender's thread rank, and the last 8 bits to the receiver's thread rank. In such way, with an extra analysis of the EP tag, we 88 can identify the ranks of the sender and the receiver in any P2P communication. As results, we a thread probes a message, it knows 89 exactly in which local queue should store the probed message. 90 91 92 With the global rank map, tag extension, and the matching-probe techniques, we are able to use any P2P communication in the endpoint 93 environment. For the collective communications, we perform a step-by-step execution and no special technique is required. The most 94 representative functions is the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. A step-by-step execution consists of 95 3 steps (not necessarily in this order): arrangement of the source data, execution of the MPI function by all 96 master/root threads, distribution or arrangement of the data among threads. 97 98 For example, if we want to perform a broadcast operation, 2 steps are needed. Firstly, the root thread, along with the master threads of 99 other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the master threads send data to threads 100 sharing the same process via local memory transfer. In another example for illustrating the \verb|MPI_Gather| function, we also need 2 101 steps. First of all, data is gathered from slave threads to the master thread or the root thread. Next, the master thread and the root 102 thread execute the \verb|MPI_Gather| operation of complete the communication. Other collective calls such as \verb|MPI_Scan|, 103 \verb|MPI_Reduce|, \verb|MPI_Scatter| \textit{etc} follow the same principle of step-by-step execution. 104 27 the performance of the climate models (client). The clients and servers are executed in parallel and communicate asynchronously. In this 28 way, the I/O peaks can be smoothed out as data fluxes are send to server constantly throughout the simulation and the time spent on data 29 writing on the server side can be overlapped completely by calculates on the client side. 30 31 \begin{figure}[ht] 32 \includegraphics[scale=0.4]{Charge1.png} 33 \includegraphics[scale=0.4]{Charge2.png} 34 \caption{On the left, each peak of computing power corresponds to the valley of memory bandwidth which shows that the computing resources 35 are alternating between calculates and I/O. ON the right, both curves are smooth which means that the computing resources have a stable 36 charge of work, either calculates or I/O.} 37 \end{figure} 38 39 40 XIOS works well with many climate simulation codes. For example, LMDZ\footnote{LMDZ is a general circulation model (or global climate model) 41 developed since the 70s at the "Laboratoire de Météorologie Dynamique", which includes various variants for the Earth and other planets 42 (Mars, Titan, Venus, Exoplanets). The 'Z' in LMDZ stands for "zoom" (and the 'LMD' is for 'Laboratoire de Météorologie Dynamique"). 43 \url{http://lmdz.lmd.jussieu.fr}}, NENO\footnote{Nucleus for European Modeling of the Ocean alias NEMO is a 44 state-of-the-art modelling framework of ocean related engines. \url{https://www.nemo-ocean.eu}}, ORCHIDEE\footnote{the land surface 45 model of the IPSL (Institut Pierre Simon Laplace) Earth System Model. \url{https://orchidee.ipsl.fr}}, and DYNAMICO\footnote{The DYNAMICO 46 project develops a new dynamical core for LMD-Z, the atmospheric general circulation model (GCM) part of IPSL-CM Earth System Model. 47 \url{http://www.lmd.polytechnique.fr/~dubos/DYNAMICO/}} all use XIOS as the output back end. M\'et\'eoFrance and MetOffice also choose XIOS 48 to manege the I/O for their models. 49 50 51 \section{Development of thread-friendly XIOS} 52 53 Although XIOS copes well with many models, there is one potential optimization in XIOS which needs to be investigated: making XIOS thread-friendly. 54 55 This topic comes along with the configuration of the climate models. Take LMDZ as example, it is designed with the 2-level parallelization scheme. To be more specific, LMDZ uses the domain decomposition method in which each sub-domain is associated with one MPI process. Inside of the sub-domain, the model also uses OpenMP derivatives to accelerate the computation. We can imagine that the sub-domain be divided into sub-sub-domain and is managed by threads. 56 57 \begin{figure}[ht] 58 \centering 59 \includegraphics[scale=0.5]{domain.pdf} 60 \caption{Illustration of the domain decomposition used in LMDZ.} 61 \end{figure} 62 63 As we know, each sub-domain, or in another word, each MPI process is a XIOS client. The data exchange between client and XIOS servers is handled by MPI communications. In order to write an output field, all threads must gather the data to the master thread who acts as MPI process in order to call MPI routines. There are two disadvantages about this method : first, we have to spend time on gathering information to the master thread which not only increases the memory use, but also implies an OpenMP barrier; second, while the master thread calls MPI routine, other threads are in the idle state thus a waster of computing resources. What we want obtain with the thread-friendly XIOS is that all threads can act like MPI processes. They can call directly the MPI routine thus no waste in memory nor in computing resources as shown in Figure \ref{fig:omp}. 64 65 \begin{figure}[ht] 66 \centering 67 \includegraphics[scale=0.6]{omp.pdf} 68 \caption{} 69 \label{fig:omp} 70 \end{figure} 71 72 There are two ways to make XIOS thread-friendly. First of all, change the structure of XIOS which demands a lot of modification is the XIOS library. Knowing that XIOS is about 100 000 lines of code, this method will be very time consuming. What's more, the modification will be local to XIOS. If we want to optimize an other code to be thread-friendly, we have to redo the modifications. The second choice is to add an extra interface to MPI in order to manage the threads. When a thread want to call an MPI routine inside XIOS, it will first pass the interface, in which the communication information will be analyzed before the MPI routine is invoked. With this method, we only need to modify a very small part of XIOS in order to make it work. What is more interesting is that the interface we created can be adjusted to suit other MPI based libraries. 73 74 75 In this project, we choose to implement an interface to handle the threads. To do so, we introduce the MPI\_endpoint which is a 76 concept proposed in the last MPI Forums and several papers have already discussed the importance of such idea and have introduced the 77 framework of the MPI\_endpoint \cite{Dinan:2013}\cite{Sridharan:2014}. The concept of an endpoint is shown by Figure \ref{fig:scheme}. In 78 the MPI\_endpoint environment, each OpenMP thread will be associated with a unique rank (global endpoint rank), an endpoint communicator, 79 and a local rank (rank inside the MPI process) which is very similar to the \verb|OMP_thread_num|. The global endpoint rank will replace the 80 role of the classic MPI rank and will be used in MPI communication calls. 81 82 83 \begin{figure}[ht] 84 \begin{center} 85 \includegraphics[scale=0.4]{scheme.png} 86 \end{center} 87 \caption{} 88 \label{fig:scheme} 89 \end{figure} 90 91 %XIOS is a library dedicated to IO management of climate code. It has a client-server pattern in which clients are in charge of computations and servers manage the reading and writing of files. The communication between clients and servers are handled by MPI. However, some of the climate models (\textit{e.g.} LMDZ) nowadays use an hybrid programming policy. Within a shared memory node, OpenMP directives are used to manage message exchanges. In such configuration, XIOS can not take full advantages of the computing resources to maximize the performance. This is because XIOS can only work with MPI processes. Before each call of XIOS routines, threads of one MPI process must gather their information to the master thread who works as an MPI process. After the call, the master thread distributes the updated information among its slave threads. As result, all slave threads have to wait while the master thread calls the XIOS routines. This introduce extra synchronization into the model and leads to not optimized performance. Aware of this situation, we need to develop a new version of XIOS (EP\_XIOS) which can work with threads, or in other words, can consider threads as they were processes. To do so, we introduce the MPI endpoints. 92 93 An other important aspect about the MPI\_endpoint interface is that each endpoints has knowledge of the ranks of other endpoints in the 94 same communicator. This knowledge is necessary because when executing an MPI communication, for example a point-to-point exchange, we need 95 to know not only the ranks of sender/receiver threads, but also the thread number of the sender/receiver threads and the MPI ranks 96 of the sender/receiver processes. This ranking information is implemented inside an map object included in the endpoint communicator 97 class. 98 99 100 101 102 %\newpage 103 104 %The MPI\_endpoint interface we implemented lies on top of an existing MPI Implementation. It consists of wrappers to all MPI functions 105 %used in XIOS. 106 107 108 In XIOS, we used the ``probe'' technique to search for arrived messages and then perform the receiving action. The principle is 109 that sender process executes the send operation as usual. However, to minimize the time spent on waiting incoming messages, the receiver 110 process calls in the first place the \verb|MPI_Probe| function to check if a message destinate to it has been published. If yes, the 111 process execute in the second place the \verb|MPI_Recv| to receive the message. If not, the receiver process can carry on with other tasks 112 or repeats the \verb|MPI_Probe| and \verb|MPI_Recv| actions if the required message is in immediate need. This technique works well in 113 the current version of XIOS. However, if we introduce threads into this mechanism, problems can occur: The incoming message is labeled by 114 the tag and receiver's MPI rank. Because threads within a process share the MPI rank, and the message probed is always available in the 115 message queue, it can lead to the problem of data race and thus the message can be received by the wrong thread. 116 117 118 To solve this problem, we introduce the ``matching-probe'' technique. The idea of the method is that each thread is equipped with a local 119 incoming message queue. Each time a thread calls an MPI function, for example \verb|MPI_Recv|, it calls firstly the \verb|MPI_Mprobe| 120 function to query the MPI incoming message with any tag and from any source. Once a message is probed, the thread gets the handle to the 121 incoming message and this specific message is erased from the MPI message queue. Then, the thread proceed the identification of the message 122 to get the destination thread's local rank and store the message handle to the local queue of the target thread. The thread repeats these 123 steps until the MPI incoming message queue is empty. Then the thread we perform the usual ``probe'' technique to query its local incoming 124 message queue to check if the required message is available. If yes, it performs the \verb|MPI_Recv| operation. With this ``matching-probe'' 125 technique, we can assure that a message is probed only once and is received by the right receiver. 126 127 128 129 Another issue needs to be clarified with this technique is that: how to identify the receiver's rank? The solution to this question is to 130 use the tag argument. In the MPI environment, a tag is an integer ranging from 0 to $2^{31}$ depending on the Implementation. We can explore 131 the large range property of the tag to store in it information about the source and destination thread ranks. In our endpoint interface, we 132 choose to limit the first 15 bits for the tag used in the classic MPI communication, the next 8 bits to store the sender thread's local 133 rank, and the last 8 bits to store the receiver thread's local rank (\textit{c.f.} Figure \ref{fig:tag}). In this way, with an extra 134 analysis of the tag, we can identify the local ranks of the sender and the receiver in any P2P communication. As results, when a thread 135 probes a message, it knows exactly in which local queue should store the probed message. 136 137 \begin{figure}[ht] 138 \centering 139 \includegraphics[scale=0.4]{tag.png} 140 \caption{}\label{fig:tag} 141 \end{figure} 142 143 In Figure \ref{fig:tag}, Tag contains the user defined value for a certain communication. MPI\_tag is computed in the endpoint interface 144 with help of the rank map and is used in the MPI calls. 145 146 \begin{figure}[ht] 147 \centering 148 \includegraphics[scale = 0.4]{sendrecv.png} 149 \caption{This figure shows the classic pattern of a P2P communication with the endpoint interface. Thread/endpoint rank 0 sends a message 150 to thread/endpoint rank 3 with tag=1. The underlying MPI function called by the sender is indeed a send for MPI rank of 1 151 and tag=65537. From the receiver's point of view, the endpoint 3 is actually receving a message from MPI rank 0 with 152 tag=65537.} 153 \label{fig:sendrecv} 154 \end{figure} 155 156 157 158 159 With the rank map, tag extension, and the matching-probe techniques, we are now able to call any P2P communication in the endpoint 160 environment. For the collective communications, we apply a step-by-step execution pattern and no special technique is required. A 161 step-by-step execution pattern consists of 3 steps (not necessarily in this order and not all steps are needed): arrangement of the source 162 data, execution of the MPI function by all master/root threads, distribution or arrangement of the resulting data among threads. 163 164 %The most representative functions of the collective communications are \verb|MPI_Gather| and \verb|MPI_Bcast|. 165 166 For example, if we want to perform a broadcast operation, only 2 steps are needed (\textit{c.f.} Figure \ref{fig:bcast}). Firstly, the root 167 thread, along with the master threads of other processes, perform the classic \verb|MPI_Bcast| operation. Secondly, the root thread, and the 168 master threads send data to other threads via local memory transfer. 169 170 \begin{figure}[ht] 171 \centering 172 \includegraphics[scale=0.3]{bcast.png} 173 \caption{} 174 \label{fig:bcast} 175 \end{figure} 176 177 Figure \ref{fig:allreduce} illustrates how the \verb|MPI_Allreduce| function is proceeded in the endpoint interface. First of all, We 178 perform a intra-process ``allreduce'' operation: source data is reduced from slave threads to the master thread via local memory transfer. 179 Next, alm master threads call the classic \verb|MPI_Allreduce| routine. Finally, all master threads send the updated reduced data to its 180 slaves via local memory transfer. 181 182 \begin{figure}[ht] 183 \centering 184 \includegraphics[scale=0.3]{allreduce.png} 185 \caption{} 186 \label{fig:allreduce} 187 \end{figure} 188 189 Other MPI routines, such as \verb|MPI_Wait|, \verb|MPI_Intercomm_create| \textit{etc.}, can be found in the technique report of the 190 endpoint interface. 191 192 \section{The multi-threaded XIOS and performce results} 193 194 The development of endpoint interface for thread-friendly XIOS library took about one year and a half. The main difficulty is the 195 co-existance of MPI processes and OpenMP threads. All MPI classes must be redefined in the endpoint interface along with all the routines. 196 The development is now available on the forge server: \url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp}. One 197 technique report is also available in which one can find more detail about how endpoint works and how the routines are implemented 198 \cite{ep:2018}. We must note that the thread-friendly XIOS library is still in the phase of optimization. It will be released in the 199 future with a stable version. 200 201 All the funcionalities of XIOS is reserved in its thread-friendly version. Single threaded code can work successfully with the new 202 version of XIOS. For multi-threaded models, some modifications are needed in order to work with the multi-threaded XIOS library. Detail can 203 be found in our technique report \cite{ep:2018}. 204 205 Even though the multi-threaded 105 206 106 207 \section{Performance of LMDZ using EP\_XIOS} … … 114 215 simulation duration settings: 1 day, 5 days, 15 days, and 31 days. 115 216 116 \begin{figure}[h ]217 \begin{figure}[ht] 117 218 \centering 118 219 \includegraphics[scale = 0.6]{LMDZ_perf.png} … … 131 232 decrease in time of 25\%. Even the 25\% may seems to be small, it is still a gain in performance with existing computing resources. 132 233 234 \section{Performance of EP\_XIOS} 235 236 workfloz\_cmip6 237 light output 238 24*8+2 239 30s - 52s 240 32 days 241 histmth with daily output 242 133 243 \section{Perspectives of EP\_XIOS} 134 244 245 246 \bibliographystyle{plain} 247 \bibliography{reference} 248 135 249 \end{document} -
XIOS/dev/branch_openmp/Note/reference.bib
r1548 r1552 6 6 7 7 @conference {Dinan:2013, 8 title = {Enabling MPI Interoperability Through Flexible Communication Endpoints},8 title = {Enabling {MPI} {I}nteroperability {T}hrough {F}lexible {C}ommunication {E}ndpoints}, 9 9 booktitle = {EuroMPI 2013}, 10 10 year = {2013}, … … 16 16 author={S. Sridharan and J. Dinan and D. D. Kalamkar}, 17 17 booktitle={SC14: International Conference for High Performance Computing, Networking, Storage and Analysis}, 18 title={Enabling Efficient Multithreaded MPI Communication through a Library-Based Implementation of MPI Endpoints},18 title={Enabling {E}fficient {M}ultithreaded {MPI} {C}ommunication {T}hrough a {L}ibrary-{B}ased {I}mplementation of {MPI} {E}ndpoints}, 19 19 year={2014}, 20 20 pages={487-498}, 21 21 month={Nov} 22 22 } 23 24 @techreport{ep:2018, 25 author={XIOS developper group}, 26 title={Note for {XIOS} {E}ndpoints}, 27 year={2018}, 28 address={\url{http://forge.ipsl.jussieu.fr/ioserver/browser/XIOS/dev/branch_openmp/Note}} 29 } -
XIOS/dev/branch_openmp/Note/tag.svg
r1548 r1552 5 5 <defs/> 6 6 <g> 7 <g latexdraw:type="rectangle" id="id 480">7 <g latexdraw:type="rectangle" id="id32"> 8 8 <rect x="661.0" y="201.0" width="178.0" height="38.0" 9 9 latexdraw:borderPos="outer" stroke-width="2.0" 10 10 stroke="black" fill="#f0d7e6"/> 11 11 </g> 12 <g latexdraw:type="rectangle" id="id 481">12 <g latexdraw:type="rectangle" id="id33"> 13 13 <rect x="221.0" y="201.0" width="258.0" height="38.0" 14 14 latexdraw:borderPos="outer" stroke-width="2.0" 15 15 stroke="black" fill="#d0f7e0"/> 16 16 </g> 17 <g latexdraw:type="rectangle" id="id 482">17 <g latexdraw:type="rectangle" id="id34"> 18 18 <rect x="481.0" y="201.0" width="178.0" height="38.0" 19 19 latexdraw:borderPos="outer" stroke-width="2.0" 20 20 stroke="black" fill="#c8e7ec"/> 21 21 </g> 22 <g latexdraw:type="rectangle" id="id 483">22 <g latexdraw:type="rectangle" id="id35"> 23 23 <rect x="221.0" y="201.0" width="618.0" height="38.0" 24 24 latexdraw:borderPos="outer" stroke-width="2.0" 25 25 stroke="black" fill="none"/> 26 26 </g> 27 <g latexdraw:type="line" id="id 484">27 <g latexdraw:type="line" id="id36"> 28 28 <line x1="480.0" x2="480.0" y1="200.0" y2="240.0" 29 29 stroke-width="2.0" stroke="black" latexdraw:rotationAngle="0.0"/> 30 30 </g> 31 <g latexdraw:type="line" id="id 485">31 <g latexdraw:type="line" id="id37"> 32 32 <line x1="660.0" x2="660.0" y1="200.0" y2="240.0" 33 33 stroke-width="2.0" stroke="black" latexdraw:rotationAngle="0.0"/> 34 34 </g> 35 <g latexdraw:type="text" id="id 486" font-family="Dialog"35 <g latexdraw:type="text" id="id38" font-family="Dialog" 36 36 fill="black" font-size="18" latexdraw:x="320.0"> 37 <text x="320.0" y="230.0"><![CDATA[ EP_tag]]></text>37 <text x="320.0" y="230.0"><![CDATA[Tag]]></text> 38 38 </g> 39 <g latexdraw:type="text" id="id4 89" font-family="Dialog"39 <g latexdraw:type="text" id="id41" font-family="Dialog" 40 40 fill="black" font-size="18" latexdraw:x="510.0"> 41 41 <text x="510.0" y="230.0"><![CDATA[src local rank]]></text> 42 42 </g> 43 <g latexdraw:type="text" id="id4 92" font-family="Dialog"43 <g latexdraw:type="text" id="id43" font-family="Dialog" 44 44 fill="black" font-size="18" latexdraw:x="690.0"> 45 45 <text x="690.0" y="230.0"><![CDATA[dst local rank]]></text> 46 46 </g> 47 <g latexdraw:type="text" id="id4 95" font-family="Dialog"47 <g latexdraw:type="text" id="id45" font-family="Dialog" 48 48 fill="black" font-size="18" latexdraw:x="100.0"> 49 49 <text x="100.0" y="230.0"><![CDATA[MPI_tag =]]></text> 50 50 </g> 51 <g latexdraw:type="text" id="id4 98" font-family="Dialog"51 <g latexdraw:type="text" id="id47" font-family="Dialog" 52 52 fill="black" font-size="18" latexdraw:x="400.0"> 53 53 <text x="400.0" y="270.0"><![CDATA[15 bits]]></text> 54 54 </g> 55 <g latexdraw:type="text" id="id 501" font-family="Dialog"55 <g latexdraw:type="text" id="id49" font-family="Dialog" 56 56 fill="black" font-size="18" latexdraw:x="590.0"> 57 57 <text x="590.0" y="270.0"><![CDATA[8 bits]]></text> 58 58 </g> 59 <g latexdraw:type="text" id="id5 04" font-family="Dialog"59 <g latexdraw:type="text" id="id51" font-family="Dialog" 60 60 fill="black" font-size="18" latexdraw:x="770.0"> 61 61 <text x="770.0" y="270.0"><![CDATA[8 bits]]></text> 62 62 </g> 63 <g latexdraw:type="text" id="id5 07" font-family="Dialog"63 <g latexdraw:type="text" id="id53" font-family="Dialog" 64 64 fill="black" font-size="18" latexdraw:x="570.0"> 65 <text x="570.0" y="290.0"><![CDATA[0 ~ 25 6]]></text>65 <text x="570.0" y="290.0"><![CDATA[0 ~ 255]]></text> 66 66 </g> 67 <g latexdraw:type="text" id="id5 10" font-family="Dialog"67 <g latexdraw:type="text" id="id56" font-family="Dialog" 68 68 fill="black" font-size="18" latexdraw:x="750.0"> 69 <text x="750.0" y="290.0"><![CDATA[0 ~ 25 6]]></text>69 <text x="750.0" y="290.0"><![CDATA[0 ~ 255]]></text> 70 70 </g> 71 <g latexdraw:type="text" id="id5 13" font-family="Dialog"71 <g latexdraw:type="text" id="id59" font-family="Dialog" 72 72 fill="black" font-size="18" latexdraw:x="370.0"> 73 <text x="370.0" y="290.0"><![CDATA[0 ~ 3276 8]]></text>73 <text x="370.0" y="290.0"><![CDATA[0 ~ 32767]]></text> 74 74 </g> 75 75 </g> … … 89 89 <latexdraw:version>2.0.8</latexdraw:version> 90 90 <latexdraw:size> 91 <latexdraw:width> 1867</latexdraw:width>92 <latexdraw:height>1 176</latexdraw:height>91 <latexdraw:width>2507</latexdraw:width> 92 <latexdraw:height>1576</latexdraw:height> 93 93 </latexdraw:size> 94 94 <latexdraw:position> … … 96 96 <latexdraw:y>24</latexdraw:y> 97 97 </latexdraw:position> 98 <latexdraw:dividerPosition>0. 8489555436529191</latexdraw:dividerPosition>98 <latexdraw:dividerPosition>0.9976067012365377</latexdraw:dividerPosition> 99 99 <latexdraw:displayCodePanel>true</latexdraw:displayCodePanel> 100 100 <latexdraw:displayXScale>true</latexdraw:displayXScale>
Note: See TracChangeset
for help on using the changeset viewer.