#2074 closed Defect (fixed)
Out of bounds error in the extended row form of lbcnfd
| Reported by: | acc | Owned by: | acc |
|---|---|---|---|
| Priority: | low | Milestone: | 2018 release-4.0 |
| Component: | OCE | Version: | trunk |
| Severity: | major | Keywords: | OPA v4.0 |
| Cc: | acc |
Description
Context
Out of bounds error in the extended row form of lbcnfd which breaks the code when running with check bounds and is a probable cause of restartability issues with ORCA2LIMPISCES when icebergs are active
Analysis
The recent reorganisation and tidying of the lbc routines has introduced an out of bounds error in the extended halo form of the generic lbcnfd routine. Currently this is only used with iceberg field exchanges across the north fold but the error is sufficient to stop the model (if running with check bounds) or to introduce a restartability error in the ORCA2LIMPISCES SETTE tests when run with active icebergs.
Recommendation
The solution is a better generalisation of the mpp_lbc_north_icb and lbc_nfd_ext_generic.h90 routines which sets and passes correctly sized arrays for any number of extra halo rows. This includes zero extra haloes so the lbc_nfd_ext_generic.h90 routine could, eventually, replace the lbc_nfd_generic.h90 set if the algorithm is extended to the 3d and 4d cases. The arrays that are set and passed are centred on the same 4 rows that the default lbcnfd uses so, for example, with 1 extra halo the array second dimension ranges from 0 to 5 with rows 1 to 4 treated in same way as the original scheme. This makes for easier checks on the operations.
That said, working with these generic, preprocessor-generated routines is not straight-forward and the best way to debug and develop this solution was to work off-line with the pre-processed routine, extracted from the lbcnfd.f90 file in BLD/ppsrc/nemo and inserted into a simple wrapper program. Two such test programs are attached to this ticket: nfdtest.f90 and nfdtest_ext.f90. The first just wraps lbc_nfd_2d (extracted from BLD/ppsrc/nemo/lbcnfd.f90) and calls it repeatedly with an 8x4 array for each grid-point type and pivot location combination. The array before and after folding is printed with extra, blanked rows above and below to make a side-by-side comparsion possible. The second test program wraps lbc_nfd_2d_ext (also from lbcnfd.f90) and makes the same calls with an 8x6 array; the middle 4 rows of which hold the same values as the default test. The side-by-side comparison of the output from these two programs is shown below:
sdiff nfdtest.out nfdtest_ext.out
Default lbcnfd (no extended rows) lbcnfd with extended row (kextj = 1)
--------------------------------------------------+------------------------------------------------------
T-pt, T-pivot, npolj= 4 T-pt, T-pivot, npolj= 4
BEFORE
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "T", 1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "T", 1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | 11 16 15 14 13 12 11 10
19 24 23 22 21 20 19 18 19 24 23 22 21 20 19 18
25 26 27 28 29 28 27 26 25 26 27 28 29 28 27 26
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
U-pt, T-pivot, npolj= 4 U-pt, T-pivot, npolj= 4
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "U", -1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "U", -1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | -10 -15 -14 -13 -12 -11 -10 -15
-18 -23 -22 -21 -20 -19 -18 -23 -18 -23 -22 -21 -20 -19 -18 -23
25 26 27 -29 29 -27 -26 32 25 26 27 -29 29 -27 -26 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
V-pt, T-pivot, npolj= 4 V-pt, T-pivot, npolj= 4
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "V", -1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "V", -1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | -3 -8 -7 -6 -5 -4 -3 -2
-11 -16 -15 -14 -13 -12 -11 -10 -11 -16 -15 -14 -13 -12 -11 -10
25 -24 -23 -22 -21 -20 -19 -18 25 -24 -23 -22 -21 -20 -19 -18
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
F-pt, T-pivot, npolj= 4 F-pt, T-pivot, npolj= 4
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "F", 1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "F", 1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | 2 7 6 5 4 3 2 7
10 15 14 13 12 11 10 15 10 15 14 13 12 11 10 15
24 23 22 21 20 19 18 32 24 23 22 21 20 19 18 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
T-pt, F-pivot, npolj= 6 T-pt, F-pivot, npolj= 6
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "T", 1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "T", 1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | 24 23 22 21 20 19 18 17
32 31 30 29 28 27 26 25 32 31 30 29 28 27 26 25
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
U-pt, F-pivot, npolj= 6 U-pt, F-pivot, npolj= 6
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
U-pt, F-pivot, npolj= 6 U-pt, F-pivot, npolj= 6
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "U", -1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "U", -1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | -23 -22 -21 -20 -19 -18 -17 -17
-31 -30 -29 -28 -27 -26 -25 -25 -31 -30 -29 -28 -27 -26 -25 -25
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
V-pt, F-pivot, npolj= 6 V-pt, F-pivot, npolj= 6
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "V", -1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "V", -1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | -16 -15 -14 -13 -12 -11 -10 -9
-24 -23 -22 -21 -20 -19 -18 -17 -24 -23 -22 -21 -20 -19 -18 -17
25 26 27 28 -28 -27 -26 -25 25 26 27 28 -28 -27 -26 -25
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
BEFORE
F-pt, F-pivot, npolj= 6 F-pt, F-pivot, npolj= 6
0 0 0 0 0 0 0 0 | 41 42 43 44 45 46 47 48
33 34 35 36 37 38 39 40 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32 25 26 27 28 29 30 31 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
CALL lbc_nfd_2d( pnorth, "F", 1.0D0) | CALL lbc_nfd_2d_ext( pnorth, "F", 1.0D0, 1)
AFTER
0 0 0 0 0 0 0 0 | 15 14 13 12 11 10 9 9
23 22 21 20 19 18 17 17 23 22 21 20 19 18 17 17
25 26 27 28 27 26 25 32 25 26 27 28 27 26 25 32
17 18 19 20 21 22 23 24 17 18 19 20 21 22 23 24
9 10 11 12 13 14 15 16 9 10 11 12 13 14 15 16
0 0 0 0 0 0 0 0 | 1 2 3 4 5 6 7 8
----------------- -----------------
Crucially the results are identical for the middle four rows in each case and operations on the extended row follow the same pattern.
The corrected and generalised code changes (OPA_SRC/LBC/lbc_nfd_ext_generic.h90 and OPA_SRC/LBC/lib_mpp.F90, only) have been committed to branch 2017/dev_merge_2017 as changeset [9467]. This changeset also reinstates the Intel compiler directive in geo2ocean.F90 which is still necessary on my test system. With these changes the out of bounds error is eliminated and all SETTE tests are passed. This includes ORCA2LIMPISCES restartability tests with active icebergs. ORCA2LIMPISCES reproducibility with icebergs is achieved over the first 80 time steps only but is ok over 1000+ time steps without icebergs.
Commit History (2)
| Changeset | Author | Time | ChangeLog |
|---|---|---|---|
| 9471 | acc | 2018-04-11T18:04:11+02:00 | Branch 2017/dev_merge_2017. Add protection around compiler directive in geo2ocean.F90 so that it is only applied for Intel compilers. See ticket #2074 |
| 9467 | acc | 2018-04-11T13:22:29+02:00 | Branch 2017/dev_merge_2017. Fix for out of bounds error with the extended form of lbcnfd. See ticket #2074 |
Attachments (2)
Change History (8)
comment:1 Changed 6 years ago by acc
comment:2 Changed 6 years ago by acc
- Resolution set to fixed
- Status changed from new to closed
comment:3 Changed 6 years ago by andmirek
- Cc acc added
!DIR$ OPTIMIZE:1 directive in geo2ocean.F90 should be protested with #if defined directive. It breaks compilation with other than Intel compiler and may not be needed. I would suggest:
#if defined __INTEL_COMPILER !DIR$ OPTIMIZE:1 ! intel formulation #endif
comment:4 Changed 6 years ago by acc
Fair enough, I always thought compiler directives where meant to be just comments to other compilers but it seems other compilers can use the same directive string with different argument syntax. The suggested protection has been inserted, the SETTE tests have been re-run successfully and the modified geo2ocean.F90 has been submitted to change set [9471]
comment:5 Changed 6 years ago by acc
In 9471:
comment:6 Changed 2 years ago by nemo
- Keywords OPA v4.0 added
In 9467: