Changes between Version 26 and Version 27 of 2018WP/Shared Actions/Status_dev_merge_2017_branch


Ignore:
Timestamp:
2018-02-21T15:22:11+01:00 (2 years ago)
Author:
nicolasmartin
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • 2018WP/Shared Actions/Status_dev_merge_2017_branch

    v26 v27  
    1 = Status of work on dev_merge_2017 branch = 
     1= Status of work on dev_merge_2017 branch 
    22 
    3 [[PageOutline]] 
     3[[PageOutline()]] 
    44 
    55Last edition on '''[[Wikinfo(changed_ts)]]''' by '''[[Wikinfo(changed_by)]]''' 
    66 
    7  
    8 [[PageOutline(3, , inline, unnumbered)]] 
    9  
    107The 2017/dev_merge_2017 branch has been created by merging all the 2017 developments during Merge party in Exeter. 
    11 This branch is expected to be back into the trunk as the prelimnary version of future NEMO release as soon as possible in 2018. 
     8This branch is expected to be back into the trunk as the preliminary version of future NEMO release as soon as possible in 2018. 
    129This page allows developer to share information and results on the on going testing work on this branch. 
    1310 
    1411''For each new subject, please follow format as below (== for title level 2) in order to have them listed at top of page, and add SOLVED in the title when it is done. 
    1512 
    16 == 16 February 2018 : METO: eORCA1 and eORCA025 occasional failures with NaNs  
     13== 16 February 2018 : METO: eORCA1 and eORCA025 occasional failures with !NaNs  
    1714 
    1815I'm getting occasional failures with a segmentation fault in icb_ground in eORCA1 and eORCA025 built from the latest dev_merge_2017 branch (after several months of integration). I think icb_ground is acting as an error trap for NaN values produced in ice_dyn_rhg_evp. I will try to trace it back further next week. (Dave Storkey). 
    1916 
    2017'''Update 21 Feb:''' This looks like some kind of instability which doesn't get picked up by the checks in stp_ctl. There is very thick ice (~100m) at a single point which I think results in a divide-by-zero in sbc_isf_div. It looks different to the instability in the the 300 timestep ORCA2LIM3PIS_ST run in SETTE, which doesn't show any abnormally thick ice.  
     18 
    2119== 12 February 2018 : ORCA2_LIM3_PISCES with icebergs is not reproducible 
    2220 
    23 == 12 February 2018 : ORCA2_LIM3_PISCES reproductibility problem 
     21== 12 February 2018 : ORCA2_LIM3_PISCES reproducibility problem 
     22 
    2423This continues the point from Feb 1rst and summarizes the work done  and the mail exchanges by Clement Bricaud, Andrew Coward, Pierre Mathiot Tomas Lovato and others. 
    2524 
    26 Thre is a problem  for now on the MPI reproducibility of ORCA2_LIM3_PISCES without icebergs on head of dev_merge_2017 branch. 
     25There is a problem  for now on the MPI reproducibility of ORCA2_LIM3_PISCES without icebergs on head of dev_merge_2017 branch. 
    2726All the following tests are made '''on ORCA2_LIM3_PISCES without icebergs''' 
    2827 
     
    3534This branch must now be investigated to see which revision causes the failure et find the error. (Now ongoing at CNRS) 
    3635''' 
     36 
    3737== 01 February 2018 
    3838 
    39 dev_merge_2017 (rev9271)  : ORCA2LIM3PIS not reproductible and crashes at 179 timesteps ( so restartabily NOK because this run 150 timesteps ) 
     39dev_merge_2017 (rev9271)  : ORCA2LIM3PIS not reproducible and crashes at 179 timesteps ( so restartabily NOK because this run 150 timesteps ) 
    4040 
    41 dev_METO_MERCATOR_2017    : ORCA2LIM3PIS     reproductible and crashes at 134 timesteps ( so restartabily NOK because this run 150 timesteps ) 
     41dev_METO_MERCATOR_2017    : ORCA2LIM3PIS     reproducible and crashes at 134 timesteps ( so restartabily NOK because this run 150 timesteps ) 
    4242 
    43 dev_METO_2017             : ORCA2LIM3PIS     reproductible and run 1500 timesteps 
     43dev_METO_2017             : ORCA2LIM3PIS     reproducible and run 1500 timesteps 
    4444 
    45 dev_MERCATOR_2017(rev8976): ORCA2LIM3PIS     reproductible and crashes at 132 timesteps( so restartabily NOK because this run 150 timesteps ) 
     45dev_MERCATOR_2017(rev8976): ORCA2LIM3PIS     reproducible and crashes at 132 timesteps( so restartabily NOK because this run 150 timesteps ) 
    4646 
    47 dev_MERCATOR_2017(rev8973): ORCA2LIM3PIS     reproductible and crashes at 132 timesteps( so restartabily NOK because this run 150 timesteps ) 
     47dev_MERCATOR_2017(rev8973): ORCA2LIM3PIS     reproducible and crashes at 132 timesteps( so restartabily NOK because this run 150 timesteps ) 
    4848 
    49 dev_MERCATOR_2017(rev8922): ORCA2LIM3PIS     reproductible and run 1500 timesteps 
     49dev_MERCATOR_2017(rev8922): ORCA2LIM3PIS     reproducible and run 1500 timesteps 
    5050 
    5151 
    5252== 30 January 2018: NOC: eORCA1 testing: Pb with Weddell Sea Polynya 
     53 
    5354George has been running an eORCA1 configuration with the dev_merge_2017 branch. The configuration runs successfully but fails after 6 years (from 1978) with issues associated with a Weddell Sea polynya. Initial impression is also that the mixed layer depths (using OSMOSIS) are shallower than in runs using the pre-merge development branch.  
    5455 
    5556== 12 January 2018: Pb in ORCA2_LIM3_OBS 
    56 Sette, as it is, is succesfull with ORCA2_LIM3_OBS on MetO machine BUT ocean output contains an E R R O R. Only one time step is present in run.stat. Error is in dia_obs, the sea-ice model is not detected by dia_obs. I (PM) added a test in sette_rpt (r9223) to catch these cases (it looks for E R R O R in ocean.output). Not perfect (we should compare the number of lines in run.stat to nitend as well) but good enough for now I think. 
     57 
     58Sette, as it is, is successful with ORCA2_LIM3_OBS on MetO machine BUT ocean output contains an E R R O R. Only one time step is present in run.stat. Error is in dia_obs, the sea-ice model is not detected by dia_obs. I (PM) added a test in sette_rpt (r9223) to catch these cases (it looks for E R R O R in ocean.output). Not perfect (we should compare the number of lines in run.stat to nitend as well) but good enough for now I think. 
    5759 
    5860== 12 January 2018: Pb ORCA2OFFPIS (SOLVED) 
     61 
    5962Sette ORCA2OFFPIS failed to run on MetO machine. It stop during the initialisation phase. I (PM) added a test on the presence of the config directory in NEMO_VALIDATION repository to catch it (r9221). 
    6063 
    6164== 12 January 2018: Pb lib_mpp ? 
     65 
    6266Dave Storkey tested the dev_merge_2017@9209. eORCA1 job currently blows up on the first timestep with this lib_mpp issue. eORCA025 job runs OK for a month. He thinks that it does look like it might be an F-pivot/T-pivot thing.  
    6367 
    6468UPDATE 15 Feb: The failure is a segmentation fault in a deallocate statement in mpp_nfd_3d_ptr in a piece of code protected by l_north_nogather=.true. If I switch ln_nnogather to .false. (default value) then it runs OK (which probably explains why George can run ORCA1). 
     69 
    6570== 11 January 2018: pb with dynspg_ts (time splitting) 
    66 Clément Rousset thinks there might have been a problem during merge of developments with dynspg_ts: looks like drag is computed twice so that energy is wrong and too big. This may be the reason for the crash of ORCA2 configuration for now after 200 timesteps (so that is is not seend through SETTE tests sopping after 150 timesteps! 
     71 
     72Clément Rousset thinks there might have been a problem during merge of developments with dynspg_ts: looks like drag is computed twice so that energy is wrong and too big. This may be the reason for the crash of ORCA2 configuration for now after 200 timesteps (so that is is not send through SETTE tests sopping after 150 timesteps! 
    6773Could Jérôme have a look at this? 
    6874 
    6975== 11 January 2017 : AGRIF not restartable nor reproducible for now with sea-ice 
     76 
    7077Clément Rousset and Simona are working on it, building a test case for that. It is an important issue to solve at first. 
    7178 
    7279'''Please add new subjects ABOVE this line''' 
     80 
    7381---- 
     82 
    7483== SETTE test results  
     84 
    7585To be completed: 
    7686