HPC-01_SebEtCie_MPI_Interface

[smasson]

Context

1) Review, simplification and cleaning of the MPI interface is needed
2) There is a few bugs:

• potential bug with the corners if nn_hls > 1 (or 2?) with point-to-point communication
• collective communications is not working with BDY (as BDY has its own exchanges)

3) There is other things to further improve:

• do not send/receive if send/receive only land points (depend on nn_hls)
• send/receive only the oce values (checking mask, depend on nn_hls)
• keep in memory the send/receive buffers (if they are not too big for ex < jpi*jpj) from one call to another -> for point-to-point put the mpi_waitall at the beginning of next call
• make sure the north pole folding does not need east-west comm. done if nogather = F. to be done if nogather = T... so we could place it where we want (no need to wait for east-west comm).
• if north pole folding: force the last row of MPI-subdomains to have jpj much smaller (the smallest value?) so they go faster and other don't wait for them

Workplan action

Wikipage: wiki:2021WP/HPC-01_SebEtCie_MPI_Interface

[smasson]

In 14313:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14314:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

With nn_hls=1 and point-to-point comm., sette tests passed but:

• ORCA2_ICE_PISCES is no more working with the icebergs
• OFF_PISCES differ with trunk@14312
• SAS_ICE differ with trunk@14312
• AGRIF differ with trunk@14312
• ORCA2_ICE_OBS: incomplete test

I will check this points...
Collective comm. are also working but I need to double check this...

Brief description of what was done and what should be done...

• review, cleaning and simplification
• CRS no more working. Large parts of obsolete code have been commente out...
• deep rewritting of MPI interface
• land-only domains are no more communicating with others even if they are not suppressed
• simplification and convergence of the point-to-point and collectives interfaces
• MPI addresses store in mpinei: an integer(8) array -> could be later integer(8,n_hlsmax) when we also test the mask to see oif we need to send oce value.
• buffer size is the same for send and receive (isizei(8), isizej(8)) could be later generalized to isize(8,2,n_hlsmax) for send receive of the 8 corners.
• collective communicators (mpi_nc_com4 and mpi_nc_com8) could later be arrays to pick-up different graph (BDY and different values of nn_hls for example)
• following collective comm, back to 1D buffers in point-to-point, so later we will be able to ajust their size and re-use the buffer if fitting and not too big (to we can keep it in memeory)

[smasson]

In 14319:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14320:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14326:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14327:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

With [14327]. All sette tests passed (except SWG as in the trunk).
Compared with trunk@14310 all files are identical except:

WORCA2_OFF_PISCES_ST  tracer.stat files are DIFFERENT (results are different after   time steps)
WORCA2_SAS_ICE_ST     run.stat    files are DIFFERENT (results are different after  4  time steps)
WICE_AGRIF_ST         run.stat    files are DIFFERENT (results are different after  2  time steps)

[smasson]

In 14331:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

• With [14331], all sette tests give the same results as in the trunk@14310. :-)
  
• This is a temporary fix as the gestion of jperio will be completely reviewed.
  

[smasson]

In 14336:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14337:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14337] passes all sette tests and gives the same results as trunk@14310

[smasson]

In 14338:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14338] passes all sette tests and gives the same results as trunk@14310

In this commit, I reduced the complexity of lbclnk and lbcnfd.

• I replaced lbc_lbk_multi by lbc_lbk -> we are now always using what was corresponding to lbc_lbk_multi even if we have only 1 array
• I strongly reduce the number of interfaces in lbclnk and lbcnfd in order to simplify and speed-up the compilation time of these routines
• I renamed some of the lbc generic routines to make the hierarchy of routines more understandable (hopefully)
• I simplified the c preprocessing in the generic routines
• I added a new namelist parameter "nn_comm" in nammpp (better name to be found...) to chose between point-to-point or neighbourhood collectives communications

[smasson]

In 14343:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14344:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14346:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14349:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14350:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

Need to modify AGRIF so it recognizes the CONTIGUOUS attributes, we have been forced to add in lbc_blk_call.h90 in [14350].

In lbc_blk_call.h90, this fortran 2008 argument is indeed needed in the TARGET declarations:

REAL(PRECISION), DIMENSION(DIMS)          , TARGET, CONTIGUOUS, INTENT(inout) ::   pt1        ! arrays on which the lbc is applied
REAL(PRECISION), DIMENSION(DIMS), OPTIONAL, TARGET, CONTIGUOUS, INTENT(inout) ::   pt2 , pt3 , pt4 , pt5 , pt6 , pt7 , pt8 , pt9, &
         &                                                                         pt10, pt11, pt12, pt13, pt14, pt15, pt16

...

REAL(PRECISION), DIMENSION(DIMS), TARGET, INTENT(inout), CONTIGUOUS ::   ptab       ! arrays on which the lbc is applied

in order compile the following line

ptab_ptr(kfld)%pt4d(1:SIZE(ptab, dim=1),1:SIZE(ptab, dim=2),1:ISZ3,1:ISZ4) => ptab

This line and CONTIGUOUS are the key tricks found by Andrew to use 4d pointer even with 2d or 3d input arrays and therefore suppress all 2d and 3d interfaces of lbc_lnk_pt2pt, lbc_lnk_neicoll, lbc_nfd, mpp_nfd and lbc_nfd_nogather.
This greatly reduces the size of lnclnk and lbcfld routines and makes the generic routines much more readable (as the only remaining cpp parameters in lbc_lnk_pt2pt, lbc_lnk_neicoll, lbc_nfd, mpp_nfd and lbc_nfd_nogather are PRECISION and MPI_TYPE).

[smasson]

In 14351:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14352:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14352] passes all sette tests and gives the same results as trunk@14310

[smasson]

In 14355:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14363:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14363] passes all sette tests and gives the same results as trunk@14310
For an unexplained reason, this optimisation is not wotking with neighbourhood collective communications (nn_comm /= 1).
In consequence (and hopefully temporary), I have been force to by-pass this optimisation when nn_comm /= 1...
This is done in init_excl_landpt in mppini.F90 line 1186:

IF( nn_comm == 1 ) THEN       ! SM: NOT WORKING FOR NEIGHBOURHOOD COLLECTIVE COMMUNICATIONS, I DON'T KNOW WHY... 
...
ENDIF

[smasson]

In 14365:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14366:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14366] passes all sette tests and gives the same results as trunk@14310

[smasson]

In 14367:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14368:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14369:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14369] passes all sette tests and gives the same results as trunk@14310

[smasson]

In 14379:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14379] requires to add a few lines in ext/FCM/lib/Fcm/BuildTask.pm (thank you Andrew!)
With this change, [14379] passes all sette tests and gives the same results as trunk@14310

ext/FCM/lib/Fcm/BuildTask.pm

@@ -124 +124 @@
 
   return unless $self->actiontype;
 
+  if ( defined $self->srcfile ) {
+    my $bname = basename($self->srcfile->src) ;
+    for my $depend (@{ $self->dependency }) {
+      if ( $bname eq $depend ) {
+       # Recursion suspected
+       return;
+      }
+    }
+  }
+
   my $uptodate     = 1;
   my $dep_uptodate = 1;

[smasson]

other developments to consider:

• fix the optimization for lnc_lbk_neicoll. As specified in ticket:2598#comment:27, suppressing communications involving only land points is not working with lnc_lbk_neicoll. I don’t know why.

• review specific lbc in BDY. This first needs a review of lsend_bdy and lrecv_bdy in bdyinit with the help of Gaston Irrmann. For example, how do we deal with corners.

• review/supress specific lbc in ICB…

• Now that all calls to lbc_lnk use what was called the "4d multi” interface, it should be possible to call lnc_lnk with arrays of different shape as soon as the first 2 dimensions (i and j) are equal. I think this is quite easy to do in lbc_lnk_pt2pt and in lbc_lnk_neicoll. We mostly have to declare ipk and ipl as 1d array and rewrite the loops as DO jf = 1, ipf ; DO jl = 1, ipl(jf) ; DO jk = 1, ipk(jk). The problem comes from the NP folding. NP folding routines should also be adapted to deal with different values of ipl(jf) and ipk(jk) -> see next point

• North-Pole Folfing: should be compliant with 3 new constrains:

(1) should work with halo size khls /= nn_hls. I think the first point is needed for ICB.
(2) should not assume that east-west communication has already been done (-> do the NP folling only with inner domain points). this suppress dependency between communications.
(3) should be able to deal with different values of ipl(jf) and ipk(jk), see previous point.

I modified lbc_nfd_generic.h90 so it is compliant with constrains 1 and 2 (I hope) but I think that mpp_nfd_generic.h90 lbc_nfd_nogather_generic.h90 are not working if khls /= nn_hls. They should be modified/updated.
I think there is a room for some optimization in the “nogather" version when looking at the neighbourgs: (1) exclude communications including only land points and (2) I think we should not consider the halos when looking for neighbourgs.

• optimize northern cores size when doing NP folding:
  

As pointed out by Jesus Labarta, the NP folding is slowing down the northern cores and quickly de-synchronize all cores which slows down everything.. A quick and simple fix/patch is to minimize the j-size of the northern domains so they compute faster and they can take a longer time to do the NP folding without slowing down the other cores.
A first optimization in this direction is already existing : we force all domain rows, except the northern one, to have jpjmax lines which minimizes the j-size of the remaining northern row of domains (depending of the domain decomposition). So today we first do the domain decomposition and next play with the potential difference between jpj and jpjmax to minimize the northern domains.
We could modify this strategy. We could, for example, first fixe the northern domains j-size at a minimal value (to be defined) and next built the domain decomposition on the remaining part of the domain. This would ensure that the j-size of the northern domains is always minimized. This is not very difficult do do once we have a good criterion do define the “minimal j-size” we want to use…

• activate in lbc_lnk the optional argument ld4only whenever it is possible to communicate only with 4 neighbourgs. Would additional options such as do only exchanges in a given direction would be helpful? If yes, this is very easy to do.

• open questions: we could also fill the buffers and send only the ocean points of the halos. Maybe this would make a différence when working with 3D arrays? This could easily be done with the pack/unpack fortran functions. The problem would be to find which mask must be used…

[smasson]

In 14403:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14403] passes all sette tests and gives the same results as trunk@14401
Note that we stil need the patch on ext/FCM/lib/Fcm/BuildTask.pm as described in comment:37

[smasson]

[14403] passes all sette tests and gives the same results as trunk@14401 including with debug options (X64_IRENE_DEBUG) of with gcc (X64_IRENE_GCC)

[smasson]

In 14417:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14430:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14431:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14432:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

In 14433:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smasson]

[14434] passes all sette tests and gives the same results as trunk@14401

[smasson]

In 14611:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found