#734 closed defect (fixed)

Stomate 4D files incomplete on Irene

Reported by: mmcgrath Owned by: luyssaert
Priority: major Milestone: ORCHIDEE 4.1
Component: Biogeochemical processes Version:
Keywords: Cc:

Description

When running r6930 on Irene (prod mode) for FG1 (analytical spinup), the default settings result in a crash in pack_output, which causes the whole job to crash every 10 years or so (the default pack period). The reason seems to be that the stomate_history_4dim file is incomplete:

FG1.CROWN.r6930_19150101_19151231_1M_stomate_history_4dim {
dimensions:

axis_nbounds = 2 ;
lon = 180 ;
lat = 90 ;
time_counter = UNLIMITED ; (0 currently)

variables:

float lat(lat) ;

lat:axis = "Y" ;
lat:standard_name = "latitude" ;
lat:long_name = "Latitude" ;
lat:units = "degrees_north" ;

float lon(lon) ;

lon:axis = "X" ;
lon:standard_name = "longitude" ;
lon:long_name = "Longitude" ;
lon:units = "degrees_east" ;

float RESOLUTION_X(lat, lon) ;

RESOLUTION_X:long_name = "E-W resolution" ;
RESOLUTION_X:units = "m" ;
RESOLUTION_X:online_operation = "once" ;
RESOLUTION_X:_FillValue = 9.96921e+36f ;
RESOLUTION_X:missing_value = 9.96921e+36f ;
RESOLUTION_X:coordinates = "" ;

float RESOLUTION_Y(lat, lon) ;

RESOLUTION_Y:long_name = "N-S resolution" ;
RESOLUTION_Y:units = "m" ;
RESOLUTION_Y:online_operation = "once" ;
RESOLUTION_Y:_FillValue = 9.96921e+36f ;
RESOLUTION_Y:missing_value = 9.96921e+36f ;
RESOLUTION_Y:coordinates = "" ;

float CONTFRAC(lat, lon) ;

CONTFRAC:long_name = "Continental fraction" ;
CONTFRAC:units = "1" ;
CONTFRAC:online_operation = "once" ;
CONTFRAC:_FillValue = 9.96921e+36f ;
CONTFRAC:missing_value = 9.96921e+36f ;
CONTFRAC:coordinates = "" ;

float Areas(lat, lon) ;

Areas:long_name = "Mesh areas" ;
Areas:units = "m2" ;
Areas:online_operation = "once" ;
Areas:_FillValue = 9.96921e+36f ;
Areas:missing_value = 9.96921e+36f ;
Areas:coordinates = "" ;

global attributes:

:name = "stomate_history_4dim" ;
:description = "Created by xios" ;
:title = "Created by xios" ;
:Conventions = "CF-1.6" ;
:timeStamp = "2020-Nov-09 07:56:41 GMT" ;
:uuid = "9f9bc0e2-6107-4d1d-bf97-b8abd054e642" ;
:LongName = "Crown diameter is now constant at 66% of height. Should improve tropical evaporation/transpiration ratio." ;

When POST tries to change an attribute, it crashes because it doesn't find the time_counter variable (or some variable...unfortunately, I no longer have the error file).

A simple fix is to suppress output of the 4dim file in stomate.card:

output_level_stomate_history_4dim = NONE

The 4dim files seem to work on obelix, but only for certain output levels (at level 3, a new z axis is introduced...at level 9, a new y axis is introduced...the problem occurs at one of these two places, despite that stomate_history seems not to care about this same issue). Therefore, there seems to be a problem with how the stomate 4dim files are written in the code, or how the variables are defined in the xml files of XIOS.

Change History (4)

comment:1 Changed 13 months ago by luyssaert

In r6819 it was already noted that the variable ROOT_PROFILE was causing some trouble in XIOS. By changing the history level in src_xml/file_def_orchidee.xml this problem was circumvented but clearly not solved. ROOT_PROFILE is written on grid_nslm_nvm_out axis

Observations:

  • When running a job with ROOT_PROFILE being the only 4d variable in the output file, ferret throws the following warning
    *** NOTE: Unspecified or unsupported ordering of axes in variable ROOT_PROFILE
    *** NOTE: The default ordering will be used
    
  • When running a job with output variables written to the grid_nvm_ncirc_out or grid_nvm_nlevelstot_out axes all seems to work correctly
  • Adding variables written to the grid_nslm_nvm_out and/or grid_ngrnd_nvm_out damages the content of the variables written on the grid_nvm_ncirc_out or grid_nvm_nlevelstot_out axes.
  • Moving ROOT_PROFILE to sechiba_4d does not result in the same problem although the variables in sechiba4d have also differently ordered axes.
Last edited 13 months ago by luyssaert (previous) (diff)

comment:2 Changed 13 months ago by luyssaert

It looks like there are two (related?) issues:
(1) libIGCM post-processing has a problem with packing the 4d files on Irene. Simply not writing the 4d file solved this problem (see above).
(2) The 4d stomate file is written but the axes of the results may be inconsistent.

The second issues comes with the following observations:

  • Inconsistent axes are observed in Ferret but not in IDL.
  • Writing a variable with inconsistent axes in stomate_4d to sechiba_4d does not result in inconsistencies in sechiba_4d.
  • Both stomate_4d and sechiba_4d contain variables with very differently ordered axes.
  • The variables causing the problems in stomate_4d were not correctly initialized (this has been fixed in r6942).

The first issue comes with the following observation:

  • output level was set to 1 but none of the variables at output level 1 had 4d (this has been fixed in r6943).
  • The default 4d output was set to NONE (r6944). First, these variables are very detailed and more suitable for debugging then for global analyzes. Second, this should solve the issue.

The changes were tested and the default configuration now runs on Irene. Changing the default setting of the history level no longer results in a crash.




Last edited 13 months ago by luyssaert (previous) (diff)

comment:3 Changed 13 months ago by luyssaert

  • Owner changed from somebody to luyssaert
  • Status changed from new to assigned

comment:4 Changed 13 months ago by luyssaert

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.