munmap_chunk(): invalid pointer in XIOS2.5

[mcastril]

Hi,

In BSC we are testing a new EC-Earth branch devoted to introduce XIOS2.5 rev1566. We have noticed some issues when removing the land processes in NEMO at the same time that some specific number of processors is used for XIOS. For example, having 285 procs in NEMO and 47 in XIOS leads to this error, which is not reproduced when only having 1 XIOS server. However, we can run with 47 XIOS servers and 288 NEMO procs.

With v2.0 we didn't have this issue and we always were running with 47XIOS servers, to fill one node with the runoff_mapper, and any number of NEMO procs (using ELPiN to remove land subdomains).

The error is the following:

>>>ignore_signals(): DR_HOOK will ignore signal#65 altogether
*** Error in `./xios_server.exe': corrupted double-linked list: 0x0000000002d228a0 ***
*** Error in `./xios_server.exe': double free or corruption (out): 0x000000000295f830 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002240890 ***
*** Error in `./xios_server.exe': double free or corruption (out): 0x00000000031a9ff0 ***
*** Error in `./xios_server.exe': double free or corruption (out): 0x0000000001c267b0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000003150d40 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002ec1810 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x000000000235b250 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x00000000023454e0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x000000000300c8e0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x00000000030d6100 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002777ba0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002ae76b0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x00000000030e4f90 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000001ecf3d0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x000000000212b330 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x00000000036ab0e0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x000000000322db70 ***
*** Error in `./xios_server.exe': double free or corruption (out): 0x00000000030e04e0 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002887f50 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000002869f10 ***
*** Error in `./xios_server.exe': munmap_chunk(): invalid pointer: 0x0000000001ccfe10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x721af)[0x2b6fb2c6c1af]
/lib64/libc.so.6(+0x77706)[0x2b6fb2c71706]
/lib64/libc.so.6(+0x78453)[0x2b6fb2c72453]

It seems it is related with some invalid deallocation. The error shows up just in the first step of the simulation, when XIOS is initializing.