= ORCHIDEE trunk rev 2425 = [[PageOutline]] All coupled simulations are done with LMDZ5 testing rev 2163 using the old physics. == Parallelism and compile mode == * ORCHIDEE gives the identical results for simulations with different number of cores MPI and/or OMP. It does not matter if the executable is compiled for pur MPI mode or for mixt MPI/OMP mode. * '''Exception in coupled mode at curie: different number of cores MPI changes the results due to LMDZ''', see details below. In offline mode it's ok. * Simulations coupled to LMDZ can now run in debug mode. The correction in LMDZ trunk was done : [http://web.lmd.jussieu.fr/trac/changeset/2158]. * The optimization mode (prod/dev/debug) changes the results in most cases. An exception is at curie in offline mode where all executables give the same results. * Sequential mode is tested with gfortran : ORCHIDEE ok. LMDZ can not compile cosp (optional for more output). ==== ADA/IDRIS ==== * Changing the number of cores MPI and/or OMP does not change the results. For example running on 23MPI=32MPI=16MPI.2OMP * Using optimization mode prod does not give the same results as optimization mode dev or debug * LMDZORChoi and LMDZORCWRR fails in debug mode for 18mpi8omp. Maybe problem with memory. ==== CURIE/TGCC ==== * Offline mode: all simulations are identical : compile mode, different number of cores MPI, pur mpi or mixt mpi_omp (but only running on 1 omp) * Coupled mode: * '''Simulations are not identical while changing the number of cores MPI also using LMDZ bucket without ORCHIDEE''' * '''Simulations are identical while changing the number of cores MPI and OMP in hybrid mode ''' * Results are different between debug and dev optimisation mode and between debug and prod. ==== local PC using gfortran ==== * LMDZ do not compile with cosp * Optimization mode prod=dev=debug with ORCHIDEE offline. Only tested prod in LMDZOR. * All tests are reproducible == Restartablilty == * Same simulation set up is reproducible * '''1+1 not = 2''' in coupled and in offline mode = Details of simulations = == Offline == {{{ ORChoi at curie : ALL OK : debug=dev=prod Different number of cores mpi or mpi+1omp give the same results. All simulations finished ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi ORChoi at ada : debug = dev not= prod Different number of cores mpi or mpi+1omp give the same results. All simulations finished ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi ORCWRR sur curie : ALL simulations ok and equal Debug simulations did not finish : 30min not enough, 1hour ok ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi_omp/32mpi_1ompn results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi/32mpin results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi ORCWRR sur ada : debug = dev not= prod Different number of cores mpi or mpi+1omp give the same results. All simulations finished ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi }}} == LMDZ without ORCHIDEE == {{{ LMDZ only(without ORCHIDEE) at curie: All simulations finished Different number of cores mpi DO NOT give the same results with mpi mode. Different number of cores mpi or omp in hybrid mode give the same results ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp LMDZ only(without ORCHIDEE) at ada: All simulations finished Different number of cores mpi or omp give the same results. ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp }}} == LMDZOR == {{{ LMDZORChoi at curie: All simulations finished Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok. Different number of cores mpi or omp in hybrid mode give the same results ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp LMDZORChoi at ada : 1 simulation not finished debug : 16mpi_8omp => after relaunch still not ok! Different number of cores mpi or omp give the same results. ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp LMDZORCWRR at curie: All simulations finished Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok. Different number of cores mpi or omp in hybrid mode give the same results ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp LMDZORCWRR at ada : 1 simulation not finish: debug 16mpi_8omp => after relanch still not ok. Pb memoire ? Different number of cores mpi or omp give the same results. ******************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp }}}