= ORCHIDEE trunk rev 2247 with LMDZ5 trunk rev 2076 = This page is dedicated to technical validation of the model. For scientific validation, see more simulations here: [wiki:testing/rev2247/simulations] [[PageOutline]] == Parallelism == * ORCHIDEE gives the identical results for simulations with different number of cores MPI and/or OMP. It does not matter if the executable is compiled for pur MPI mode or for mixt MPI/OMP mode. * At curie, compile options must be updated, see below * Simulations coupled to LMDZ do not succeed in debug mode (and not dev at ada). This seems be due to following change on LMDZ : [http://web.lmd.jussieu.fr/trac/changeset/2011/LMDZ5/trunk/libf/phylmd/coefcdrag.F90]. The problem seems to be only in the first time-step. * The executable compiled in pure MPI and in mixt MPI_OMP gives the same results if the same optimization mode is used (prod/dev/debug) * Sequential mode is tested with gfortran : simulations are reproducible ==== ADA/IDRIS ==== * Compilation en mode prod, dev and debug are successful for pure mpi and mixt mpi_omp. * '''Coupled mode in dev and debug mode: simulations stops''' * Changing the number of cores MPI and/or OMP does not change the results. For example running on 23MPI=32MPI=16MPI.2OMP * The executable compiled in pure MPI and in hybrid MPI_OMP gives the same results if the same optimization mode is used (prod/dev/debug) * Using optimization mode prod does not give the same results as optimization mode dev or debug * Using optimization mode dev or debug gives the same results ==== CURIE/TGCC ==== * Compilation en mode prod, dev and debug are successful for pure mpi and mixt mpi_omp. * Offline mode: all simulations are identical : compile mode, different number of cores MPI, pure mpi or mixt mpi_omp (but only running on 1 omp) * Coupled mode: * '''Simulations are not identical while changing the number of cores MPI and or OMP''' also without ORCHIDEE using LMDZ bucket * In debug mode: simulations stops (stops when ORCHIDEE is activated, ok with bucket) * Coupled mode with modifications in compile options for LMDZ as in commit [http://web.lmd.jussieu.fr/trac/changeset/2108]: * '''Simulations are identical while changing the number of cores MPI and or OMP''' * '''Simulations are identical between prod, dev and debug compile options''' * Simulations still stops in debug mode for LMDZOR simulations ==== local PC using gfortran ==== * Compiling with gfortran for sequential mode : prod, dev and debug is ok. * Simulations are reproductive * Tests are done with ORCHIDEE offline Choisnel and CWRR, for prod, dev and debug mode * Tests are done in coupled mode for prod mode == Restartablilty == * Same simulation set up is reproducible (see above) * '''1+1 not = 2''' in coupled and in offline mode == Performance == === LMDZOR at curie === Following performances are observed using '''LMDZ !r2076 modified by adding -fp-model precise''' in compile options, with AP(old physics) and NPv3.2(CMIP5 version of new physics) : || Resolution || LMDZ physics || Nb cores || Consumption per 10 year || Elapsed time per 10 year || || 96x95x39 || AP || 32MPI*1OMP || 740hCpu || 23h || || 96x95x39 || NPv3.2 || 32MPI*1OMP || 1220hCpu || 38h || || 96x95x39 || AP || 32MPI*2OMP || 850hCpu || 13h || || 96x95x39 || AP || 32MPI*4OMP || 1100hCpu || 8h30min || Note: * Using Choisnel or CWRR do not change the run time in coupled mode. * No difference in time using executable MPI or MPI_OMP on the same number of cores MPI and 1OMP. === ORCHIDEE offline global at curie with CRU-NCEP === || Resolution || Hydrology || Nb cores MPI || Consumption per 10 year || Elapsed time for 10 year || set up with spinup analytic || || 1deg || Choisnel || 32MPI || 52h || 1.5h || no || || 1deg || CWRR || 32MPI || 65h || 2h || no || || 0.5deg || Choisnel || 32MPI || 180h || 5.5h || no || || 0.5deg || CWRR || 32MPI || 270h || 8.5h || no || || 0.5deg || Choisnel || 64MPI || 240h || 3.8h || no || || 0.5deg || CWRR || 64MPI || 340h || 5.3h || no || || 0.5deg || Choisnel || 64MPI || 310h || 4.8h || yes || || 0.5deg || CWRR || 64MPI || 460h || 7.2h || yes || Note: * The first year, starting without restart files is more than 2 times longer for CWRR than the following years. For Choisnel it is about 1.5 times longer. * The 2 last lines are using the experiment set up with spinup analytic. This set up also activates less output. The cpu consumption is higher for an unkknown reason. To be investigated. ---- ---- == Details of simulations == === Default set up === LMDZ is always run with it's old physics package (called AP). ORCHIDEE is always run activating Stomate. Tests are done with both Choisnel and CWRR hydrology. === ADA/IDRIS === The model is installed by the trusting here : /workgpfs/rech/lmd/rlmd979/TRUSTING/ORCHIDEE/RUNDIR/trunk_rev2247_20140717T1820/modipsl {{{ ****** LMDZORChoi en prod OK (les test bigtest=n) *********************************************************************************************** results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_4omp.1 results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_8omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/32mpi_2omp ****** LMDZORCWRR en prod tout OK *********************************************************************************************** results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_4omp.1 results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_8omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/32mpi_2omp ****** LMDZORChoi en dev not OK *********************************************************************************************** results.out.dev_mpi:Following 0 DIRECTORIES finished: results.out.dev_mpi:Following 2 DIRECTORIES DID NOT FINISH: 23mpi 32mpi ****** ORChoi prod/dev/debug OK ****** Compilation debug=dev mais pas debug=prod ou dev=prod ****** *********************************************************************************************** results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/10mpi_1omp.1 results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.2 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.3 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/32mpi_1omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.2 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.3 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/32mpi_1omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/32mpi_1omp ****** ORCWRR OK *********************************************************************************************** results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/10mpi_1omp.1 results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.2 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.3 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/32mpi_1omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.2 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.3 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/32mpi_1omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/32mpi_1omp }}} === CURIE/TGCC === The model is installed by the trusting here : /ccc/scratch/cont003/dsm/p86ghatt/TRUSTING/ORCHIDEE/RUNDIR/trunk_rev2247_20140717T1824/modipsl {{{ ****** LMDZORCWRR ****** debug : simulations did not finish ****** prod and dev : all simulations finished, results differs when changing the number of cores MPI/OMP. ****** Simulations are reproductable at same number of cores. ************************************************************************************************************* results.out.debug_mpi:Following 0 DIRECTORIES finished: results.out.debug_mpi:Following 2 DIRECTORIES DID NOT FINISH: 23mpi 32mpi results.out.debug_mpi_omp:Following 0 DIRECTORIES finished: results.out.debug_mpi_omp:Following 3 DIRECTORIES DID NOT FINISH: 16mpi_8omp 32mpi_1omp 32mpi_2omp results.out.dev_mpi:Following 2 DIRECTORIES finished: 23mpi 32mpi results.out.dev_mpi:Following 0 DIRECTORIES DID NOT FINISH: results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:Following 3 DIRECTORIES finished: 16mpi_8omp 32mpi_1omp 32mpi_2omp results.out.dev_mpi_omp:Following 0 DIRECTORIES DID NOT FINISH: results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_8omp NOT = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_8omp NOT = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:Following 2 DIRECTORIES finished: 23mpi 32mpi results.out.prod_mpi:Following 0 DIRECTORIES DID NOT FINISH: results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:Following 6 DIRECTORIES finished: 16mpi_4omp.1 16mpi_4omp.2 16mpi_4omp.3 16mpi_8omp 32mpi_1omp 32mpi_2omp results.out.prod_mpi_omp:Following 0 DIRECTORIES DID NOT FINISH: results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_4omp.1 NOT = prod_mpi_omp/16mpi_8omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_4omp.1 NOT = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/32mpi_2omp ****** LMDZORChoi ****** debug: No simulations finished ****** prod and dev: simulations finished, results differs when changing the number of cores MPI/OMP. ****** Simulations are reprodusable at same number of cores. ************************************************************************************************************* results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = prod_mpi_omp/16mpi_1omp results.out.debug_mpi:Following 0 DIRECTORIES finished: results.out.debug_mpi:Following 2 DIRECTORIES DID NOT FINISH: 23mpi 32mpi results.out.debug_mpi_omp:Following 0 DIRECTORIES finished: results.out.debug_mpi_omp:Following 3 DIRECTORIES DID NOT FINISH: 16mpi_8omp 32mpi_1omp 32mpi_2omp results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_8omp NOT = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_8omp NOT = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:Following 2 DIRECTORIES finished: 23mpi 32mpi results.out.prod_mpi:Following 0 DIRECTORIES DID NOT FINISH: results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_1omp = prod_mpi_omp/16mpi_2omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/16mpi_4omp.1 results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/16mpi_8omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/1mpi_1omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/32mpi_2omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/32mpi_4omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/32mpi_8omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/4mpi_1omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_1omp NOT = prod_mpi_omp/8mpi_1omp ****** ORChoi ****** All simulations finished. All simulations are identic. ************************************************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/16mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi/16mpi results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/10mpi_1omp.1 results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.2 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.3 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/32mpi_1omp results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/18mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/1mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/20mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/23mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/32mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/4mpi results.out.dev_mpi:DIRECTORIES OK : dev_mpi/16mpi = dev_mpi/8mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.2 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.3 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/16mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/1mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/2mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/4mpi_1omp results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/8mpi_1omp results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/18mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/1mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/20mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/23mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/32mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/4mpi results.out.prod_mpi:DIRECTORIES OK : prod_mpi/16mpi = prod_mpi/8mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/16mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/1mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/2mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/4mpi_1omp results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/8mpi_1omp ****** ORCWRR ****** All simulation that finished gives the same results : compile mode, number of cores MPI and or OMP ****** debug: Not all simulations finished. This is probably due to a cpu time exit. The option CHECK_CWRR is very time consuming. ****** prod: OK. All simulations finished. Same results changing the number of cores MPI/OMP. Pur mpi gives the same results as mixt mpi_omp ****** Simulations are reprodusable at same number of cores. ************************************************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = dev_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.1 results.out.cross:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = prod_mpi/23mpi results.out.cross:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.1 results.out.debug_mpi:Following 0 DIRECTORIES finished: results.out.debug_mpi:Following 2 DIRECTORIES DID NOT FINISH: 23mpi 32mpi results.out.debug_mpi_omp:Following 4 DIRECTORIES finished: 10mpi_1omp.1 10mpi_1omp.2 10mpi_1omp.3 32mpi_1omp results.out.debug_mpi_omp:Following 0 DIRECTORIES DID NOT FINISH: results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.2 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/10mpi_1omp.3 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/10mpi_1omp.1 = debug_mpi_omp/32mpi_1omp results.out.dev_mpi:Following 2 DIRECTORIES finished: 23mpi 32mpi results.out.dev_mpi:Following 0 DIRECTORIES DID NOT FINISH: results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi results.out.dev_mpi_omp:Following 4 DIRECTORIES finished: 10mpi_1omp.1 10mpi_1omp.2 10mpi_1omp.3 32mpi_1omp results.out.dev_mpi_omp:Following 0 DIRECTORIES DID NOT FINISH: results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.2 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/10mpi_1omp.3 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/10mpi_1omp.1 = dev_mpi_omp/32mpi_1omp results.out.prod_mpi:Following 2 DIRECTORIES finished: 23mpi 32mpi results.out.prod_mpi:Following 0 DIRECTORIES DID NOT FINISH: results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi results.out.prod_mpi_omp:Following 4 DIRECTORIES finished: 10mpi_1omp.1 10mpi_1omp.2 10mpi_1omp.3 32mpi_1omp results.out.prod_mpi_omp:Following 0 DIRECTORIES DID NOT FINISH: results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/10mpi_1omp.3 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/10mpi_1omp.1 = prod_mpi_omp/32mpi_1omp ****** LMDZ with bucket (no ORCHIDEE) ****** All simulation finished ****** Simulations with same number of cores are identical ****** Simulations are not the same while changing the number of cores MPI and/or OMP ****** Debug simulations seems to be identical while changing the number of cores MPI and/or OMP ************************************************************************************************************* results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_4omp.1 results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_4omp.1 results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_4omp.1 results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_4omp.1 results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_4omp.1 results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_4omp.1 results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_4omp.1 = debug_mpi_omp/16mpi_4omp.2 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_4omp.1 = debug_mpi_omp/16mpi_4omp.3 results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_4omp.1 = debug_mpi_omp/16mpi_8omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_4omp.1 = debug_mpi_omp/32mpi_1omp results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_4omp.1 = debug_mpi_omp/32mpi_2omp results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_4omp.1 = dev_mpi_omp/16mpi_4omp.2 results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_4omp.1 = dev_mpi_omp/16mpi_4omp.3 results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_4omp.1 NOT = dev_mpi_omp/16mpi_8omp results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_4omp.1 NOT = dev_mpi_omp/32mpi_1omp results.out.dev_mpi_omp:DIRECTORIES DIFFER : dev_mpi_omp/16mpi_4omp.1 NOT = dev_mpi_omp/32mpi_2omp results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.2 results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_4omp.1 = prod_mpi_omp/16mpi_4omp.3 results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_4omp.1 NOT = prod_mpi_omp/16mpi_8omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_4omp.1 NOT = prod_mpi_omp/32mpi_1omp results.out.prod_mpi_omp:DIRECTORIES DIFFER : prod_mpi_omp/16mpi_4omp.1 NOT = prod_mpi_omp/32mpi_2omp }}} === gfortran/ private pclinux laptop === The model is installed here /home/jgipsl/TRUSTING/ORCHIDEE/RUNDIR/trunk_rev2247_20140812T1536/modipsl