excessive memory use in iom_def

[smasson]

Context

In iom_def, we define the structures that will be used to read/write NetCDF files.
To do so, we use some parameter that are hard coded :

INTEGER, PARAMETER, PUBLIC ::   jpmax_files  = 100  !: maximum number of simultaneously opened file
INTEGER, PARAMETER, PUBLIC ::   jpmax_vars   = 1200 !: maximum number of variables in one file
INTEGER, PARAMETER, PUBLIC ::   jpmax_dims   =  4   !: maximum number of dimensions for one variable

We use them to define a structure, called “file_descriptor”, that will contain some of the NetCDF header informations.
Among them we have several arrays:

CHARACTER(LEN=32), DIMENSION(jpmax_vars)  ::   cn_var   !: names of the variables
INTEGER, DIMENSION(jpmax_vars)            ::   nvid     !: id of the variables
INTEGER, DIMENSION(jpmax_vars)            ::   ndims    !: number of dimensions of the variables
LOGICAL, DIMENSION(jpmax_vars)            ::   luld     !: variable using the unlimited dimension
INTEGER, DIMENSION(jpmax_dims,jpmax_vars) ::   dimsz    !: size of variables dimensions 
REAL(kind=wp), DIMENSION(jpmax_vars)      ::   scf      !: scale_factor of the variables
REAL(kind=wp), DIMENSION(jpmax_vars)      ::   ofs      !: add_offset of the variables

Analysis

In i4r8 precision, this represents ~(jpmax_dims/2+3) real arrays of jpmax_vars points.

We next define an array of this structure:

TYPE(file_descriptor), DIMENSION(jpmax_files), PUBLIC ::   iom_file !: array containing the info for all opened files

This is equivalent to a total number of real8 points of ~ (jpmax_dims/2+3) * jpmax_vars * jpmax_files = 5 * 1200 * 100
If I made no mistakes, this is the equivalent of:

• 1 3D array with jpi = 90, jpj = 90 and jpk = 75
• 9 3D arrays with jpi = 30, jpj = 30 and jpk = 75
• 81 3D arrays with jpi = 10, jpj = 10 and jpk = 75

Recommendation

We should review the way we define this structure array...

• I think the large value for jpmax_vars comes from the 4D arrays that are written/read as different 3D arrays. => we could accept 4D arrays definition to avoid this.
• jpmax_files = 100 looks a large number to me...
• use allocatable variables would be better, adapting jpmax_vars to each file (easy in read mode, more difficult in write mode?)