ORCHIDEE trunk rev 2425

All coupled simulations are done with LMDZ5 testing rev 2163 using the old physics.

Parallelism and compile mode

• ORCHIDEE gives the identical results for simulations with different number of cores MPI and/or OMP. It does not matter if the executable is compiled for pur MPI mode or for mixt MPI/OMP mode.
  • Exception in coupled mode at curie: different number of cores MPI changes the results due to LMDZ, see details below. In offline mode it's ok.
• Simulations coupled to LMDZ can now run in debug mode. The correction in LMDZ trunk was done : ​http://web.lmd.jussieu.fr/trac/changeset/2158.
• The optimization mode (prod/dev/debug) changes the results in most cases. An exception is at curie in offline mode where all executables give the same results.
• Sequential mode is tested with gfortran : ORCHIDEE ok. LMDZ can not compile cosp (optional for more output).

ADA/IDRIS

• Changing the number of cores MPI and/or OMP does not change the results. For example running on 23MPI=32MPI=16MPI.2OMP
• Using optimization mode prod does not give the same results as optimization mode dev or debug
• LMDZORChoi and LMDZORCWRR fails in debug mode for 18mpi8omp. Maybe problem with memory.

CURIE/TGCC

• Offline mode: all simulations are identical : compile mode, different number of cores MPI, pur mpi or mixt mpi_omp (but only running on 1 omp)
• Coupled mode:
  • Simulations are not identical while changing the number of cores MPI also using LMDZ bucket without ORCHIDEE
  • Simulations are identical while changing the number of cores MPI and OMP in hybrid mode
  • Results are different between debug and dev optimisation mode and between debug and prod.

local PC using gfortran

• LMDZ do not compile with cosp
• Optimization mode prod=dev=debug with ORCHIDEE offline. Only tested prod in LMDZOR.
• All tests are reproducible

Restartablilty

• Same simulation set up is reproducible
• 1+1 not = 2 in coupled and in offline mode

Details of simulations

Offline

ORChoi at curie : ALL OK : debug=dev=prod 
       	   	       	    Different number of cores mpi or mpi+1omp give the same results. 
		 	    All simulations finished
*******************************************************************************

results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi



ORChoi at ada : debug = dev not= prod  
       	      	 Different number of cores mpi or mpi+1omp give the same results. 
		 All simulations finished
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi



ORCWRR sur curie : ALL simulations ok and equal
		   Debug simulations did not finish : 30min not enough, 1hour ok
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi_omp/32mpi_1ompn
results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi/32mpin
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi



ORCWRR sur ada : debug = dev not= prod  
       	       	 Different number of cores mpi or mpi+1omp give the same results. 
		 All simulations finished
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi

LMDZ without ORCHIDEE

LMDZ only(without ORCHIDEE) at curie: All simulations finished
       	       	    Different number of cores mpi DO NOT give the same results with mpi mode.
       	       	    Different number of cores mpi or omp in hybrid mode give the same results
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp


LMDZ only(without ORCHIDEE) at ada: All simulations finished
       	       	  Different number of cores mpi or omp give the same results. 
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp

LMDZOR

LMDZORChoi at curie: All simulations finished
       	       	    Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok. 
       	       	    Different number of cores mpi or omp in hybrid mode give the same results
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp



LMDZORChoi at ada : 1 simulation not finished debug : 16mpi_8omp => after relaunch still not ok!
       	       	  Different number of cores mpi or omp give the same results. 
*******************************************************************************

results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp


LMDZORCWRR at curie: All simulations finished
       	       	    Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok. 
       	       	    Different number of cores mpi or omp in hybrid mode give the same results
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp




LMDZORCWRR at ada : 1 simulation not finish: debug 16mpi_8omp => after relanch still not ok. Pb memoire ?
       	       	  Different number of cores mpi or omp give the same results. 
*******************************************************************************
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi
results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp
results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp

results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi
results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp
results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp
results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp
results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp
results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp