Opened 7 months ago
Last modified 7 months ago
#191 new defect
Request for information on netcdf4 chunking controls
| Reported by: | acc | Owned by: | ymipsl |
|---|---|---|---|
| Priority: | major | Component: | XIOS |
| Version: | 2.5 | Keywords: | |
| Cc: |
Description
XIOS3 is likely to create more opportunities for single file output even from larger configurations. Without the need to post-process output, users will look to control chunking choices. There are attributes that can be set in the XML but the net result isn't obvious. This is a request for a clearer explanation of the algorithm that is used and clarification of the inheritance levels of these attributes.
My confusion is best illustrated with an example. For a eORCA025 configuration (x=1440, y=1206, z=75), the default settings give chunksizes of:
uos:_ChunkSizes = 1, 1206, 1440 ; uo:_ChunkSizes = 1, 3, 1206, 1440 ;
for xyt and xyzt fields respectively in all output files. This is consistent with a default chunking_blocksize_target=20MB
But the user may wish to have more chunks in the horizontal without increasing the vertical chunk factor.
Here is what I've tried:
grep chunk *.xml axis_def_nemo.xml: <axis id="deptht" long_name="Vertical T levels" unit="m" positive="down" chunking_weight="12.0" /> axis_def_nemo.xml: <axis id="deptht300" axis_ref="deptht" chunking_weight="12.0" > axis_def_nemo.xml: <axis id="depthu" long_name="Vertical U levels" unit="m" positive="down" chunking_weight="12.0" /> axis_def_nemo.xml: <axis id="depthv" long_name="Vertical V levels" unit="m" positive="down" chunking_weight="12.0" /> axis_def_nemo.xml: <axis id="depthw" long_name="Vertical W levels" unit="m" positive="down" chunking_weight="12.0" /> axis_def_nemo.xml: <axis id="ncatice" long_name="Ice category" unit="1" chunking_weight="1.0" /> domain_def_nemo.xml: <domain id="grid_T" long_name="grid T" chunking_weight_i="1.0" chunking_weight_j="1.0" /> domain_def_nemo.xml: <domain id="grid_T_inner" long_name="grid T inner" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_U" long_name="grid U" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_U_inner" long_name="grid U inner" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_V" long_name="grid V" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_V_inner" long_name="grid V inner" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_W" long_name="grid W" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_W_inner" long_name="grid W inner" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_F" long_name="grid F" chunking_weight_i="1.0" chunking_weight_j="1.0"/> domain_def_nemo.xml: <domain id="grid_F_inner" long_name="grid F inner" chunking_weight_i="1.0" chunking_weight_j="1.0"/> field_def_nemo-ice.xml: <field_group id="SBC" chunking_blocksize_target="3.0" > <!-- time step automatically defined based on nn_fsbc --> field_def_nemo-oce.xml: <field_group id="all_ocean" chunking_blocksize_target="3.0">
where the the field_group definitions are placed around all other field_group and field definitions in the respective field_def xmls. This almost works but has inconsistent results:
grid_T and grid_W files: monthly means: xyt : _ChunkSizes = 1, 603, 720 ; xyzt: _ChunkSizes = 1, 3, 603, 480 ; 5 day means: xyt : _ChunkSizes = 1, 603, 720 ; grid_U and grid_V files: monthly means: xyt : _ChunkSizes = 1, 1206, 1440 ; xyzt: _ChunkSizes = 1, 1, 1206, 1440 ; 5 day means: xyt : _ChunkSizes = 1, 603, 720 ; icemod files: monthly means: xyt : _ChunkSizes = 1, 603, 720 ; xynt: _ChunkSizes = 1, 3, 603, 480 ; 5 day means: xyt : _ChunkSizes = 1, 603, 720 ;
and was largely arrived at by trial and error. How exactly are the various chunking weights used to arrive at the final sizes?
Any clues as to why the grid_U and grid_V chunks should end up different for the monthly means? Is my assumption, that setting the chunking_blocksize_target in an empcompassing field_group means it will be inherited by all the fields, correct?
Attachments (10)
Change History (35)
comment:1 Changed 7 months ago by jderouillat
comment:2 Changed 7 months ago by jderouillat
For the inconsistency you mentioned, chunking_blocksize_target is an attribute as another, it should be inherited by all fields of the group.
For now, I didn't reproduce this problem. Could you provide your xml files ?
comment:3 Changed 7 months ago by acc
Attached the working set for the ocean at least. The ice files are separate; but it is the monthly ocean fields that have the inconsistency.
comment:4 Changed 7 months ago by acc
C++ is not my forte, but to match your explanation, should this:
if (userChunkingWeights[i]>0.)
{
countChunkingDims++;
normalizingWeight = userChunkingWeights[i];
}
be
if (userChunkingWeights[i]>0.)
{
countChunkingDims++;
normalizingWeight += userChunkingWeights[i];
}
at line 474 of src/io/onecdf4.cpp?
comment:5 Changed 7 months ago by acc
..and I'm really struggling to see how this implements the strategy (onecdf4.cpp):
std::vector<double>::iterator itChunkingRatios = chunkingRatioPerDims.begin();
//itId = dim.rbegin();
double correctionFromPreviousDim = 1.;
for (vector<StdSize>::reverse_iterator itDim = dimsizes.rbegin(); itDim != dimsizes.rend(); ++itDim, ++itChunkingRatios, ++itId)
{
*itChunkingRatios *= correctionFromPreviousDim;
correctionFromPreviousDim = 1;
if (*itChunkingRatios > 1) // else target larger than size !
{
StdSize dimensionSize = *itDim;
//info(0) << *itId << " " << *itDim << " " << *itChunkingRatios << " " << (*itDim)/(*itChunkingRatios) << endl;
*itDim = ceil( *itDim / ceil(*itChunkingRatios) );
correctionFromPreviousDim = *itChunkingRatios/ ((double)dimensionSize/(*itDim));
}
}
int storageType = (0 == dimSize) ? NC_CONTIGUOUS : NC_CHUNKED;
CNetCdfInterface::defVarChunking(grpid, varid, storageType, &dimsizes[0]);
After all that work, the chunksizes are not passed to defVarChunking? But setting the weights clearly impacts the chunksizes. More cpp magic?
comment:6 Changed 7 months ago by jderouillat
For normalizingWeight += userChunkingWeights[i];, not necessary. It's just a normalization, weight are relative one to the other.
For cpp magic, don't worry, the chunk is passed :
std::vector<double>::iterator itChunkingRatios = chunkingRatioPerDims.begin();
for (vector<StdSize>::reverse_iterator itDim = dimsizes.rbegin(); itDim != dimsizes.rend(); ++itDim, ++itChunkingRatios, ++itId)
// itDim is an iterator on the record sizes (dimsizes)
// itChunkingRatios is an iterator on chunkRatio_per_dimension
// the ++ in the for loop below indicates that we go the next elements of the 2 vectors of dimsizes and chunkingRatioPerDims
*itDim = ceil( *itDim / ceil(*itChunkingRatios) ); // kind of dimsizes( itDim ) updated
}
comment:7 Changed 7 months ago by acc
OK, a bit of cpp sleight of hand and my mistaken preconception that dimsizes were the invariant sizes of the dataset.
Thanks for the explanation. I've been able to recreate the scheme in the attached excel spreadsheet which will take some of the trial and error out of the process. At least if the user has a preference for chunk sizes they can use the spreadsheet to preview what different combinations of the weights and target will give them. It ought to be a web-app but my Java is even worse than my C++!
Just the mystery as to why the U and V monthly means turned out different remains, otherwise this ticket can be closed.
comment:8 follow-up: ↓ 10 Changed 7 months ago by jderouillat
A web-app is a good idea, we'll keep that in mind.
(dimsizes is not the invariant in this function, it is a local variable in which the invariant sizes are copied suppressing the unlimited dimension).
Concerning monthly U/V, I didn't see any differences between grid_U and grid_T treatment, or between 5d and 1m in the XML ...
For 1 month :
<file_definition type="one_file" compression_level="1" name="MEANS_OUT/@expname@_@freq@_@startdate@_@enddate@" sync_freq="1mo" min_digits="4">
<file_group id="1m" output_freq="1mo" output_level="10" split_freq="1mo" enabled=".TRUE." > <!-- real monthly files →
<file id="file61" mode="write" writer="t1mwriter" gatherer="t1mgatherer" using_server2="true" name_suffix="_grid_T" description="ocean T grid variables" >
<field field_ref="e3t" />
<field field_ref="sst" name="tos" />
<file id="file62" mode="write" writer="u1mwriter" gatherer="u1mgatherer" using_server2="true" name_suffix="_grid_U" description="ocean U grid variables" >
<field field_ref="e3u" />
<field field_ref="ssu" name="uos" />
For 5 days :
<file_definition type="one_file" compression_level="1" name="MEANS_OUT/@expname@_@freq@_@startdate@_@enddate@" sync_freq="1mo" min_digits="4">
<file_group id="5d" output_freq="5d" split_freq="1mo" output_level="10" enabled=".TRUE.">
<file id="file81" mode="write" writer="t5dwriter" gatherer="t5dgatherer" using_server2="true" name_suffix="_grid_T" description="ocean T grid variables" >
<field field_ref="sst" name="tos" />
<file id="file82" mode="write" writer="u5dwriter" gatherer="u5dgatherer" using_server2="true" name_suffix="_grid_U" description="ocean U grid variables" >
<field field_ref="ssu" name="uos" />
With symmetric field/grid/domain definitions :
<field_definition level="1" prec="4" operation="average" enabled=".TRUE." default_value="1.e20" > <!-- time step automaticaly defined -->
<field_group id="all_ocean" chunking_blocksize_target="3.0">
<field_group id="grid_T" grid_ref="grid_T_2D" >
<field id="sst" long_name="..." standard_name="..." unit="degC" />
<field_group id="grid_U" grid_ref="grid_U_2D">
<field id="ssu" long_name="..." unit="m/s" />
<grid id="grid_T_2D" >
<domain domain_ref="grid_T" />
<grid id="grid_U_2D" >
<domain domain_ref="grid_U" />
<domain id="grid_T" long_name="grid T" chunking_weight_i="1.0" chunking_weight_j="1.0" />
<domain id="grid_U" long_name="grid U" chunking_weight_i="1.0" chunking_weight_j="1.0"/>
comment:9 Changed 7 months ago by acc
For completeness, here is a simple python chunk calculator that just needs some input functionality to be really useful:
import numpy as np
isize=1440
jsize=1206
ksize=75
target=3
wi=1.0
wj=1.0
wk=12.0
prec=4
print(" XYZ domain size : ",int(isize),"x",int(jsize),"x",int(ksize))
print(" Target chunksize : ",int(target),"MB, FP precision: ",int(prec))
print(" i- j- and k- weights: ",wi, wj, wk)
#
rec3d=isize*jsize*ksize*prec
rec2d=isize*jsize*prec
targmb=target*1024*1024
chunkratio3d=rec3d/targmb
chunkratio2d=rec2d/targmb
#
# 4D chunks (xyzt). t is the record dimension and always has a chunksize of 1
#
norm3d=wi+wj+wk
prodwgts3d=wi*wj*wk/(norm3d**3)
rcnt3d=1.0/3.0
#
ratio_dim_i_3d=(wi/norm3d)*((chunkratio3d/prodwgts3d)**rcnt3d)
ratio_dim_j_3d=(wj/norm3d)*((chunkratio3d/prodwgts3d)**rcnt3d)
ratio_dim_k_3d=(wk/norm3d)*((chunkratio3d/prodwgts3d)**rcnt3d)
#
chk3d_i=np.ceil(isize/np.ceil(ratio_dim_i_3d))
resid=ratio_dim_i_3d/(isize/chk3d_i)
chk3d_j=np.ceil(jsize/np.ceil(resid*ratio_dim_j_3d))
resid=ratio_dim_j_3d/(jsize/chk3d_j)
chk3d_k=np.ceil(ksize/np.ceil(resid*ratio_dim_k_3d))
#
print(" 4D XYZT chunk sizes: ",int(chk3d_i),int(chk3d_j),int(chk3d_k),int(1))
#
# 3D chunks (xyt). t is the record dimension and always has a chunksize of 1
#
norm2d=wi+wj
prodwgts2d=wi*wj/(norm2d**2)
rcnt2d=1.0/2.0
#
ratio_dim_i_2d=(wi/norm2d)*((chunkratio2d/prodwgts2d)**rcnt2d)
ratio_dim_j_2d=(wj/norm2d)*((chunkratio2d/prodwgts2d)**rcnt2d)
#
chk2d_i=np.ceil(isize/np.ceil(ratio_dim_i_2d))
resid=ratio_dim_i_2d/(isize/chk2d_i)
chk2d_j=np.ceil(jsize/np.ceil(resid*ratio_dim_j_2d))
print(" 3D XYT chunk sizes : ",int(chk2d_i),int(chk2d_j),int(1))
Currently, it is hardwired with the eORCA025 example settings:
python3 chunk_calc.py XYZ domain size : 1440 x 1206 x 75 Target chunksize : 3 MB, FP precision: 4 i- j- and k- weights: 1.0 1.0 12.0 4D XYZT chunk sizes: 480 603 3 1 3D XYT chunk sizes : 720 603 1
comment:10 in reply to: ↑ 8 Changed 7 months ago by acc
Replying to jderouillat:
Concerning monthly U/V, I didn't see any differences between grid_U and grid_T treatment, or between 5d and 1m in the XML ...
Neither could I; which makes the persistence of this difference so confusing. It is still there, for example:
function ncc() { ncdump -h -s "$1" | grep Chunk | grep e3 ;}
ncc eORCA025_RK3_1m_19590101_19590228_grid_T_195901-195901.nc
e3t:_ChunkSizes = 1, 3, 603, 480 ;
ncc eORCA025_RK3_1m_19590101_19590228_grid_U_195901-195901.nc
e3u:_ChunkSizes = 1, 1, 1206, 1440 ;
ncc eORCA025_RK3_1m_19590101_19590228_grid_V_195901-195901.nc
e3v:_ChunkSizes = 1, 1, 1206, 1440 ;
ncc eORCA025_RK3_1m_19590101_19590228_grid_W_195901-195901.nc
e3w:_ChunkSizes = 1, 3, 603, 480 ;
The chunk sizes for U and V don't match the values you would get without setting any of the weights or block size target either, (that would give: 1, 3, 1206, 1440). Very strange.
comment:11 Changed 7 months ago by acc
I've added an INFO line to onecdf4.cpp at line 476:
info(0) << name << " " << countChunkingDims << " MYINFO " << endl;
The results may give a clue; looks like some of the U and V fields are not registering dimensions to be chunked (e.g. e3v, e3u):
grep MYINFO x*.out xios_server_0041.out:-> info : tos 2 MYINFO xios_server_0041.out:-> info : sos 2 MYINFO xios_server_0041.out:-> info : zos 2 MYINFO xios_server_0041.out:-> info : zad_imp_loc 2 MYINFO xios_server_0041.out:-> info : zad_imp_hCmax 2 MYINFO xios_server_0041.out:-> info : zad_imp_vCmax 2 MYINFO xios_server_0041.out:-> info : mldkz5 2 MYINFO xios_server_0042.out:-> info : e3u 1 MYINFO xios_server_0042.out:-> info : uos 0 MYINFO xios_server_0042.out:-> info : uo 1 MYINFO xios_server_0042.out:-> info : tauuo 0 MYINFO xios_server_0042.out:-> info : uocetr_eff 1 MYINFO xios_server_0042.out:-> info : vozomatr 1 MYINFO xios_server_0042.out:-> info : sozohetr 0 MYINFO xios_server_0042.out:-> info : sozosatr 0 MYINFO xios_server_0042.out:-> info : aeiu_2d 0 MYINFO xios_server_0042.out:-> info : e3v 1 MYINFO xios_server_0042.out:-> info : vos 0 MYINFO xios_server_0042.out:-> info : vo 1 MYINFO xios_server_0042.out:-> info : tauvo 0 MYINFO xios_server_0042.out:-> info : vocetr_eff 1 MYINFO xios_server_0042.out:-> info : vomematr 1 MYINFO xios_server_0042.out:-> info : somehetr 0 MYINFO xios_server_0042.out:-> info : somesatr 0 MYINFO xios_server_0042.out:-> info : aeiv_2d 0 MYINFO xios_server_0043.out:-> info : e3u 1 MYINFO xios_server_0043.out:-> info : uos 0 MYINFO xios_server_0043.out:-> info : uo 1 MYINFO xios_server_0043.out:-> info : tauuo 0 MYINFO xios_server_0043.out:-> info : uocetr_eff 1 MYINFO xios_server_0043.out:-> info : vozomatr 1 MYINFO xios_server_0043.out:-> info : sozohetr 0 MYINFO xios_server_0043.out:-> info : sozosatr 0 MYINFO xios_server_0043.out:-> info : aeiu_2d 0 MYINFO xios_server_0043.out:-> info : e3v 1 MYINFO xios_server_0043.out:-> info : vos 0 MYINFO xios_server_0043.out:-> info : vo 1 MYINFO xios_server_0043.out:-> info : tauvo 0 MYINFO xios_server_0043.out:-> info : vocetr_eff 1 MYINFO xios_server_0043.out:-> info : vomematr 1 MYINFO xios_server_0043.out:-> info : somehetr 0 MYINFO xios_server_0043.out:-> info : somesatr 0 MYINFO xios_server_0043.out:-> info : aeiv_2d 0 MYINFO xios_server_0044.out:-> info : uos 2 MYINFO xios_server_0044.out:-> info : vos 2 MYINFO xios_server_0045.out:-> info : e3t 3 MYINFO xios_server_0045.out:-> info : thetao 3 MYINFO xios_server_0045.out:-> info : so 3 MYINFO xios_server_0045.out:-> info : tos 2 MYINFO xios_server_0045.out:-> info : sos 2 MYINFO xios_server_0045.out:-> info : zos 2 MYINFO xios_server_0045.out:-> info : tosstd 2 MYINFO xios_server_0045.out:-> info : zosstd 2 MYINFO xios_server_0045.out:-> info : sstdcy 2 MYINFO xios_server_0045.out:-> info : mldkz5 2 MYINFO xios_server_0045.out:-> info : mldr10_1 2 MYINFO xios_server_0045.out:-> info : mldr10_1dcy 2 MYINFO xios_server_0045.out:-> info : sbt 2 MYINFO xios_server_0045.out:-> info : heatc 2 MYINFO xios_server_0045.out:-> info : saltc 2 MYINFO xios_server_0045.out:-> info : wfo 2 MYINFO xios_server_0045.out:-> info : qsr_oce 2 MYINFO xios_server_0045.out:-> info : qns_oce 2 MYINFO xios_server_0045.out:-> info : qt_oce 2 MYINFO xios_server_0045.out:-> info : sfx 2 MYINFO xios_server_0045.out:-> info : taum 2 MYINFO xios_server_0045.out:-> info : windsp 2 MYINFO xios_server_0045.out:-> info : precip 2 MYINFO xios_server_0045.out:-> info : snowpre 2 MYINFO xios_server_0045.out:-> info : e3w 3 MYINFO xios_server_0045.out:-> info : wo 3 MYINFO xios_server_0045.out:-> info : difvho 3 MYINFO xios_server_0045.out:-> info : vovematr 3 MYINFO xios_server_0045.out:-> info : av_wave 3 MYINFO xios_server_0045.out:-> info : bn2 3 MYINFO xios_server_0045.out:-> info : bflx_iwm 3 MYINFO xios_server_0045.out:-> info : pcmap_iwm 2 MYINFO xios_server_0045.out:-> info : emix_iwm 3 MYINFO xios_server_0045.out:-> info : av_ratio 3 MYINFO xios_server_0046.out:-> info : e3t 3 MYINFO xios_server_0046.out:-> info : thetao 3 MYINFO xios_server_0046.out:-> info : so 3 MYINFO xios_server_0046.out:-> info : tos 2 MYINFO xios_server_0046.out:-> info : sos 2 MYINFO xios_server_0046.out:-> info : zos 2 MYINFO xios_server_0046.out:-> info : tosstd 2 MYINFO xios_server_0046.out:-> info : zosstd 2 MYINFO xios_server_0046.out:-> info : sstdcy 2 MYINFO xios_server_0046.out:-> info : mldkz5 2 MYINFO xios_server_0046.out:-> info : mldr10_1 2 MYINFO xios_server_0046.out:-> info : mldr10_1dcy 2 MYINFO xios_server_0046.out:-> info : sbt 2 MYINFO xios_server_0046.out:-> info : heatc 2 MYINFO xios_server_0046.out:-> info : saltc 2 MYINFO xios_server_0046.out:-> info : wfo 2 MYINFO xios_server_0046.out:-> info : qsr_oce 2 MYINFO xios_server_0046.out:-> info : qns_oce 2 MYINFO xios_server_0046.out:-> info : qt_oce 2 MYINFO xios_server_0046.out:-> info : sfx 2 MYINFO xios_server_0046.out:-> info : taum 2 MYINFO xios_server_0046.out:-> info : windsp 2 MYINFO xios_server_0046.out:-> info : precip 2 MYINFO xios_server_0046.out:-> info : snowpre 2 MYINFO xios_server_0046.out:-> info : e3w 3 MYINFO xios_server_0046.out:-> info : wo 3 MYINFO xios_server_0046.out:-> info : difvho 3 MYINFO xios_server_0046.out:-> info : vovematr 3 MYINFO xios_server_0046.out:-> info : av_wave 3 MYINFO xios_server_0046.out:-> info : bn2 3 MYINFO xios_server_0046.out:-> info : bflx_iwm 3 MYINFO xios_server_0046.out:-> info : pcmap_iwm 2 MYINFO xios_server_0046.out:-> info : emix_iwm 3 MYINFO xios_server_0046.out:-> info : av_ratio 3 MYINFO xios_server_0047.out:-> info : simsk 2 MYINFO xios_server_0047.out:-> info : snvolu 2 MYINFO xios_server_0047.out:-> info : sithic 2 MYINFO xios_server_0047.out:-> info : sivolu 2 MYINFO xios_server_0047.out:-> info : siconc 2 MYINFO xios_server_0047.out:-> info : sivelu 2 MYINFO xios_server_0047.out:-> info : sivelv 2 MYINFO xios_server_0047.out:-> info : simsk 2 MYINFO xios_server_0047.out:-> info : simsk05 2 MYINFO xios_server_0047.out:-> info : simsk15 2 MYINFO xios_server_0047.out:-> info : snvolu 2 MYINFO xios_server_0047.out:-> info : sithic 2 MYINFO xios_server_0047.out:-> info : sivolu 2 MYINFO xios_server_0047.out:-> info : siconc 2 MYINFO xios_server_0047.out:-> info : sisali 2 MYINFO xios_server_0047.out:-> info : siapnd 2 MYINFO xios_server_0047.out:-> info : sivpnd 2 MYINFO xios_server_0047.out:-> info : sst_m 2 MYINFO xios_server_0047.out:-> info : sss_m 2 MYINFO xios_server_0047.out:-> info : sitemp 2 MYINFO xios_server_0047.out:-> info : sntemp 2 MYINFO xios_server_0047.out:-> info : sittop 2 MYINFO xios_server_0047.out:-> info : sitbot 2 MYINFO xios_server_0047.out:-> info : sitsni 2 MYINFO xios_server_0047.out:-> info : sivelu 2 MYINFO xios_server_0047.out:-> info : sivelv 2 MYINFO xios_server_0047.out:-> info : sivelo 2 MYINFO xios_server_0047.out:-> info : utau_ai 2 MYINFO xios_server_0047.out:-> info : vtau_ai 2 MYINFO xios_server_0047.out:-> info : utau_oi 2 MYINFO xios_server_0047.out:-> info : vtau_oi 2 MYINFO xios_server_0047.out:-> info : sidive 2 MYINFO xios_server_0047.out:-> info : sishea 2 MYINFO xios_server_0047.out:-> info : sistre 2 MYINFO xios_server_0047.out:-> info : normstr 2 MYINFO xios_server_0047.out:-> info : sheastr 2 MYINFO xios_server_0047.out:-> info : qt_oce_ai 2 MYINFO xios_server_0047.out:-> info : qt_atm_oi 2 MYINFO xios_server_0047.out:-> info : qtr_ice_top 2 MYINFO xios_server_0047.out:-> info : qtr_ice_bot 2 MYINFO xios_server_0047.out:-> info : qt_ice 2 MYINFO xios_server_0047.out:-> info : qsr_ice 2 MYINFO xios_server_0047.out:-> info : qns_ice 2 MYINFO xios_server_0047.out:-> info : qemp_ice 2 MYINFO xios_server_0047.out:-> info : albedo 2 MYINFO xios_server_0047.out:-> info : hfxcndtop 2 MYINFO xios_server_0047.out:-> info : hfxcndbot 2 MYINFO xios_server_0047.out:-> info : hfxsensib 2 MYINFO xios_server_0047.out:-> info : sfxice 2 MYINFO xios_server_0047.out:-> info : vfxice 2 MYINFO xios_server_0047.out:-> info : vfxsnw 2 MYINFO xios_server_0047.out:-> info : simskcat 3 MYINFO xios_server_0047.out:-> info : snthicat 3 MYINFO xios_server_0047.out:-> info : siconcat 3 MYINFO xios_server_0047.out:-> info : sithicat 3 MYINFO xios_server_0047.out:-> info : sisalcat 3 MYINFO xios_server_0047.out:-> info : sitemcat 3 MYINFO xios_server_0047.out:-> info : sntemcat 3 MYINFO xios_server_0047.out:-> info : sitopcat 3 MYINFO
comment:12 Changed 7 months ago by jderouillat
Could you send the Netcdf header (ncdump -h ....nc) of the U field file ?
Before the algorithm that I described, we search for the domain/axis associated to the field (the chunking weights are stored in it, not in the field). If the names are not correctly identified, dimensions will not be considered by the chunking.
comment:13 Changed 7 months ago by acc
ncdump -h eORCA025_RK3_1m_19590101_19590131_grid_U_195901-195901.nc
netcdf eORCA025_RK3_1m_19590101_19590131_grid_U_195901-195901 {
dimensions:
axis_nbounds = 2 ;
x_grid_U = 1440 ;
y_grid_U = 1206 ;
depthu = 75 ;
x_grid_T = 1440 ;
y_grid_T = 1206 ;
time_counter = UNLIMITED ; // (1 currently)
variables:
float depthu(depthu) ;
depthu:name = "depthu" ;
depthu:long_name = "Vertical U levels" ;
depthu:units = "m" ;
depthu:positive = "down" ;
depthu:bounds = "depthu_bounds" ;
float depthu_bounds(depthu, axis_nbounds) ;
depthu_bounds:units = "m" ;
float nav_lat_grid_T(y_grid_T, x_grid_T) ;
nav_lat_grid_T:standard_name = "latitude" ;
nav_lat_grid_T:long_name = "Latitude" ;
nav_lat_grid_T:units = "degrees_north" ;
float nav_lon_grid_T(y_grid_T, x_grid_T) ;
nav_lon_grid_T:standard_name = "longitude" ;
nav_lon_grid_T:long_name = "Longitude" ;
nav_lon_grid_T:units = "degrees_east" ;
double time_centered(time_counter) ;
time_centered:standard_name = "time" ;
time_centered:long_name = "Time axis" ;
time_centered:calendar = "gregorian" ;
time_centered:units = "seconds since 1900-01-01 00:00:00" ;
time_centered:time_origin = "1900-01-01 00:00:00" ;
time_centered:bounds = "time_centered_bounds" ;
double time_centered_bounds(time_counter, axis_nbounds) ;
double time_counter(time_counter) ;
time_counter:axis = "T" ;
time_counter:standard_name = "time" ;
time_counter:long_name = "Time axis" ;
time_counter:calendar = "gregorian" ;
time_counter:units = "seconds since 1900-01-01 00:00:00" ;
time_counter:time_origin = "1900-01-01 00:00:00" ;
time_counter:bounds = "time_counter_bounds" ;
double time_counter_bounds(time_counter, axis_nbounds) ;
float e3u(time_counter, depthu, y_grid_U, x_grid_U) ;
e3u:standard_name = "cell_thickness" ;
e3u:long_name = "U-cell thickness" ;
e3u:units = "m" ;
e3u:online_operation = "average" ;
e3u:interval_operation = "3600 s" ;
e3u:interval_write = "1 month" ;
e3u:cell_methods = "time: mean (interval: 3600 s)" ;
e3u:_FillValue = 1.e+20f ;
e3u:missing_value = 1.e+20f ;
e3u:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float uos(time_counter, y_grid_U, x_grid_U) ;
uos:long_name = "ocean surface current along i-axis" ;
uos:units = "m/s" ;
uos:online_operation = "average" ;
uos:interval_operation = "3600 s" ;
uos:interval_write = "1 month" ;
uos:cell_methods = "time: mean (interval: 3600 s)" ;
uos:_FillValue = 1.e+20f ;
uos:missing_value = 1.e+20f ;
uos:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float uo(time_counter, depthu, y_grid_U, x_grid_U) ;
uo:standard_name = "sea_water_x_velocity" ;
uo:long_name = "ocean current along i-axis" ;
uo:units = "m/s" ;
uo:online_operation = "average" ;
uo:cell_methods = "time: mean (thickness weighted)" ;
uo:interval_operation = "1 month" ;
uo:interval_write = "1 month" ;
uo:_FillValue = 1.e+20f ;
uo:missing_value = 1.e+20f ;
uo:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float tauuo(time_counter, y_grid_T, x_grid_T) ;
tauuo:standard_name = "surface_downward_x_stress" ;
tauuo:long_name = "Wind Stress along i-axis" ;
tauuo:units = "N/m2" ;
tauuo:online_operation = "average" ;
tauuo:interval_operation = "3600 s" ;
tauuo:interval_write = "1 month" ;
tauuo:cell_methods = "time: mean (interval: 3600 s)" ;
tauuo:_FillValue = 1.e+20f ;
tauuo:missing_value = 1.e+20f ;
tauuo:coordinates = "time_centered nav_lat_grid_T nav_lon_grid_T" ;
float uocetr_eff(time_counter, depthu, y_grid_U, x_grid_U) ;
uocetr_eff:standard_name = "ocean_volume_x_transport" ;
uocetr_eff:long_name = "Effective ocean transport along i-axis" ;
uocetr_eff:units = "m3/s" ;
uocetr_eff:online_operation = "average" ;
uocetr_eff:interval_operation = "3600 s" ;
uocetr_eff:interval_write = "1 month" ;
uocetr_eff:cell_methods = "time: mean (interval: 3600 s)" ;
uocetr_eff:_FillValue = 1.e+20f ;
uocetr_eff:missing_value = 1.e+20f ;
uocetr_eff:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float vozomatr(time_counter, depthu, y_grid_U, x_grid_U) ;
vozomatr:standard_name = "ocean_mass_x_transport" ;
vozomatr:long_name = "Ocean Mass X Transport" ;
vozomatr:units = "kg/s" ;
vozomatr:online_operation = "average" ;
vozomatr:interval_operation = "3600 s" ;
vozomatr:interval_write = "1 month" ;
vozomatr:cell_methods = "time: mean (interval: 3600 s)" ;
vozomatr:_FillValue = 1.e+20f ;
vozomatr:missing_value = 1.e+20f ;
vozomatr:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float sozohetr(time_counter, y_grid_U, x_grid_U) ;
sozohetr:standard_name = "ocean_heat_x_transport" ;
sozohetr:long_name = "ocean eulerian heat transport along i-axis" ;
sozohetr:units = "W" ;
sozohetr:online_operation = "average" ;
sozohetr:interval_operation = "3600 s" ;
sozohetr:interval_write = "1 month" ;
sozohetr:cell_methods = "time: mean (interval: 3600 s)" ;
sozohetr:_FillValue = 1.e+20f ;
sozohetr:missing_value = 1.e+20f ;
sozohetr:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float sozosatr(time_counter, y_grid_U, x_grid_U) ;
sozosatr:standard_name = "ocean_salt_x_transport" ;
sozosatr:long_name = "ocean eulerian salt transport along i-axis" ;
sozosatr:units = "1e-3*kg/s" ;
sozosatr:online_operation = "average" ;
sozosatr:interval_operation = "3600 s" ;
sozosatr:interval_write = "1 month" ;
sozosatr:cell_methods = "time: mean (interval: 3600 s)" ;
sozosatr:_FillValue = 1.e+20f ;
sozosatr:missing_value = 1.e+20f ;
sozosatr:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
float aeiu_2d(time_counter, y_grid_U, x_grid_U) ;
aeiu_2d:long_name = " surface u-EIV coefficient" ;
aeiu_2d:units = "m2/s" ;
aeiu_2d:online_operation = "average" ;
aeiu_2d:interval_operation = "3600 s" ;
aeiu_2d:interval_write = "1 month" ;
aeiu_2d:cell_methods = "time: mean (interval: 3600 s)" ;
aeiu_2d:_FillValue = 1.e+20f ;
aeiu_2d:missing_value = 1.e+20f ;
aeiu_2d:coordinates = "time_centered nav_lat_grid_U nav_lon_grid_U" ;
// global attributes:
:name = "MEANS_OUT/eORCA025_RK3_1m_19590101_19590131_grid_U" ;
:description = "ocean U grid variables" ;
:title = "ocean U grid variables" ;
:Conventions = "CF-1.6" ;
:timeStamp = "2023-Sep-19 13:27:27 GMT" ;
:uuid = "f7641010-adec-4136-aeb7-b0820bb5baf6" ;
}
comment:14 Changed 7 months ago by acc
Ok, so it is the mix of grids in those files that is the cause. If I take out the utau (tauuo) and vtau (tauvo) fields from the respective files , then the chunking works consistently. It was a user error to include those fields there, since they are on the T-grid, but probably not the last time someone is going to do that. Should we expect it to cope with mixed grid content?
comment:15 Changed 7 months ago by acc
Another question related to chunking: What is the relationship (if any) between compression_level as a file attribute or as a field attribute? I've been setting the former but it hasn't been triggering any dataset compression. Dataset compression (as evidenced by the _DeflateLevel attribute in the output netcdf file) is not triggered until I set compression_level as a field attribute. Is this the expected behaviour?
comment:16 Changed 7 months ago by jderouillat
If no compression_level has been set at the field level, it should look at the file level.
It seems that I introduced a bug in july working on more compression features (parallel, lossy).
Thank you for the bug report
comment:17 Changed 7 months ago by jderouillat
Should be fix now on the XIOS3 trunk with 2577
comment:18 Changed 7 months ago by acc
Yes, can confirm that 2566 with the additional one-line fix at 2577 now triggers dataset compression with compression_level set only as a file attribute. Futher testing with an ORCA2 set-up revealed an error in the spreadsheet and python chunk calculators which weren't applying the conditional check for chunking ratios > 1. Attaching updated versions...
comment:19 Changed 7 months ago by jderouillat
Thank you for your calculators !
Concerning the mix of grids with chunking, a fix is proposed in 2579.
comment:20 Changed 7 months ago by acc
No good. Something in that fix is causing the test to hang at start-up.
Meanwhile, I've documented progress so far in a NEMO issue which is useful as a cross-reference.
comment:21 Changed 7 months ago by acc
..looks like it is the embedded include of mpi.hpp inside file.hpp that causes the hang. If I comment out the first include of mpi.hpp, it runs....and correctly chunks all the fields -even with the mix of grids in a file.
-
src/io/onetcdf4.cpp
svn diff src/io/onetcdf4.cpp
3 3 #include "onetcdf4.hpp" 4 4 #include "onetcdf4_plugin.hpp" 5 5 #include "group_template.hpp" 6 #include "mpi.hpp"6 //#include "mpi.hpp" 7 7 #include "netcdf.hpp" 8 8 #include "netCdfInterface.hpp" 9 9 #include "netCdfException.hpp"
comment:22 Changed 7 months ago by ymipsl
What kind of hang do you have ? Any error message or other ?
comment:23 Changed 7 months ago by acc
Apologies, it seems a response I thought I'd posted didn't register. This hang appears to be an intermittent issue on our cluster and commenting out the mpi.hpp was mere coincidence. The trunk@2579 will run but crashes out soon after starting to write the first set of monthly means:
terminate called after throwing an instance of 'xios::CException' forrtl: error (76): Abort trap signal Image PC Routine Line Source xios_server.exe 00000000013E72CB for__signal_handl Unknown Unknown libpthread-2.17.s 00002B8E5AC72630 Unknown Unknown Unknown libc-2.17.so 00002B8E5B1B7387 gsignal Unknown Unknown libc-2.17.so 00002B8E5B1B8A78 abort Unknown Unknown libstdc++.so.6.0. 00002B8E58C5EEBA Unknown Unknown Unknown libstdc++.so.6.0. 00002B8E58C6A9FA Unknown Unknown Unknown libstdc++.so.6.0. 00002B8E58C6AA65 Unknown Unknown Unknown libstdc++.so.6.0. 00002B8E58C6ACF8 Unknown Unknown Unknown xios_server.exe 000000000114EB5A Unknown Unknown Unknown xios_server.exe 00000000011E6F97 MPI_Comm_dup Unknown Unknown libhdf5.so.200.1. 00002B8E5EB96A8D H5_mpi_comm_dup Unknown Unknown libhdf5.so.200.1. 00002B8E5EA0B534 Unknown Unknown Unknown libhdf5.so.200.1. 00002B8E5EA16B8E Unknown Unknown Unknown libhdf5.so.200.1. 00002B8E5EA16A99 H5P_set Unknown Unknown libhdf5.so.200.1. 00002B8E5EBA8785 H5Pset_fapl_mpio Unknown Unknown libnetcdf.so.19 00002B8E587CA352 NC4_create Unknown Unknown libnetcdf.so.19 00002B8E58739414 NC_create Unknown Unknown libnetcdf.so.19 00002B8E58735DDF nc_create_par Unknown Unknown xios_server.exe 000000000130FB51 Unknown Unknown Unknown xios_server.exe 00000000011A81F0 Unknown Unknown Unknown xios_server.exe 00000000011A2506 Unknown Unknown Unknown xios_server.exe 0000000001158761 Unknown Unknown Unknown xios_server.exe 00000000011586D7 Unknown Unknown Unknown xios_server.exe 0000000000EB6BED Unknown Unknown Unknown xios_server.exe 0000000000EB5344 Unknown Unknown Unknown xios_server.exe 0000000000EC51AF Unknown Unknown Unknown xios_server.exe 00000000011151DE Unknown Unknown Unknown xios_server.exe 00000000011AD8D1 Unknown Unknown Unknown xios_server.exe 00000000011ADB9A Unknown Unknown Unknown xios_server.exe 00000000012C310A Unknown Unknown Unknown xios_server.exe 00000000011151DE Unknown Unknown Unknown xios_server.exe 00000000011AD8D1 Unknown Unknown Unknown xios_server.exe 00000000011ADB9A Unknown Unknown Unknown xios_server.exe 00000000011F4EAD Unknown Unknown Unknown xios_server.exe 0000000000E9F139 Unknown Unknown Unknown xios_server.exe 0000000000E9E957 Unknown Unknown Unknown xios_server.exe 000000000112979B Unknown Unknown Unknown xios_server.exe 0000000001128E04 Unknown Unknown Unknown xios_server.exe 0000000001124E28 Unknown Unknown Unknown xios_server.exe 000000000095A621 Unknown Unknown Unknown xios_server.exe 00000000011EB368 Unknown Unknown Unknown libstdc++.so.6.0. 00002B8E58C95814 Unknown Unknown Unknown libpthread-2.17.s 00002B8E5AC6AEA5 Unknown Unknown Unknown libc-2.17.so 00002B8E5B27FB0D clone Unknown Unknown forrtl: error (76): Abort trap signal
comment:24 Changed 7 months ago by acc
Some more information on the crashes with XIOS3-trunk (now testing with rev 2582). Four of the xios_server_XXXX.err files are complaining:
In file "mpi_tools.cpp", function "static void xios::CCommTrack::registerComm(const int &)", line 17 -> Communicator already allocated :
and a gdb traceback on one of the core dumps shows:
(gdb) where
#0 0x00002ad522844387 in raise () from /lib64/libc.so.6
#1 0x00002ad522845a78 in abort () from /lib64/libc.so.6
#2 0x00000000013ede56 in for.signal_handler ()
#3 <signal handler called>
#4 0x00002ad522844387 in raise () from /lib64/libc.so.6
#5 0x00002ad522845a78 in abort () from /lib64/libc.so.6
#6 0x00002ad5202ebeba in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#7 0x00002ad5202f79fa in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#8 0x00002ad5202f7a65 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:58
#9 0x00002ad5202f7cf8 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x1a4f8e0 <typeinfo for xios::CException>,
dest=0xe89990 <xios::CException::~CException()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:95
#10 0x000000000115084a in xios::CCommTrack::registerComm(int const&) ()
#11 0x00000000011eac47 in MPI_Comm_dup ()
#12 0x00002ad526223a8d in H5_mpi_comm_dup () from /opt/software/eb/software/HDF5/1.12.1-iimpi-2021b/lib/libhdf5.so.200
#13 0x00002ad526098534 in ?? () from /opt/software/eb/software/HDF5/1.12.1-iimpi-2021b/lib/libhdf5.so.200
#14 0x00002ad5260a3b8e in ?? () from /opt/software/eb/software/HDF5/1.12.1-iimpi-2021b/lib/libhdf5.so.200
#15 0x00002ad5260a3a99 in H5P_set () from /opt/software/eb/software/HDF5/1.12.1-iimpi-2021b/lib/libhdf5.so.200
#16 0x00002ad526235785 in H5Pset_fapl_mpio () from /opt/software/eb/software/HDF5/1.12.1-iimpi-2021b/lib/libhdf5.so.200
#17 0x00002ad51fe57352 in NC4_create () from /opt/software/eb/software/netCDF/4.8.1-iimpi-2021b/lib/libnetcdf.so.19
#18 0x00002ad51fdc6414 in NC_create () from /opt/software/eb/software/netCDF/4.8.1-iimpi-2021b/lib/libnetcdf.so.19
#19 0x00002ad51fdc2ddf in nc_create_par () from /opt/software/eb/software/netCDF/4.8.1-iimpi-2021b/lib/libnetcdf.so.19
#20 0x00000000013159c1 in xios::CNetCdfInterface::createPar(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int, int, int&) ()
#21 0x00000000011aa940 in xios::CONetCDF4::initialize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, int const*, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
()
#22 0x00000000011a4c56 in xios::CONetCDF4::CONetCDF4(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, int const*, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#23 0x000000000115a201 in xios::CNc4DataOutput::CNc4DataOutput(xios::CFile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, int, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#24 0x000000000115a177 in xios::CNc4DataOutput::CNc4DataOutput(xios::CFile*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, int, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#25 0x0000000000eb7d0d in xios::CFile::createHeader() ()
#26 0x0000000000eb6464 in xios::CFile::checkWriteFile() ()
#27 0x0000000000ec62cf in xios::CFileWriterStoreFilter::onInputReady(std::vector<std::shared_ptr<xios::CDataPacket>, std::allocator<std::shared_ptr<xios::CDataPacket> > >) ()
#28 0x00000000011162fe in xios::CInputPin::setInput(unsigned long, std::shared_ptr<xios::CDataPacket>) ()
#29 0x00000000011b0021 in xios::COutputPin::deliverOuput(std::shared_ptr<xios::CDataPacket>) ()
#30 0x00000000011b02ea in xios::COutputPin::onOutputReady(std::shared_ptr<xios::CDataPacket>) ()
#31 0x00000000012c8f7a in xios::CFilter::onInputReady(std::vector<std::shared_ptr<xios::CDataPacket>, std::allocator<std::shared_ptr<xios::CDataPacket> > >) ()
#32 0x00000000011162fe in xios::CInputPin::setInput(unsigned long, std::shared_ptr<xios::CDataPacket>) ()
#33 0x00000000011b0021 in xios::COutputPin::deliverOuput(std::shared_ptr<xios::CDataPacket>) ()
#34 0x00000000011b02ea in xios::COutputPin::onOutputReady(std::shared_ptr<xios::CDataPacket>) ()
#35 0x00000000011f944d in xios::CServerFromClientSourceFilter::streamData(xios::CEventServer&) ()
#36 0x0000000000ea0259 in xios::CField::recvUpdateData(xios::CEventServer&) ()
#37 0x0000000000e9fa77 in xios::CField::dispatchEvent(xios::CEventServer&) ()
#38 0x00000000011b908f in xios::CP2pContextServer::dispatchEvent(xios::CEventServer&) ()
#39 0x00000000011b8d04 in xios::CP2pContextServer::processEvents(bool) ()
#40 0x00000000011b76e9 in xios::CP2pContextServer::eventLoop(bool) ()
#41 0x000000000095b251 in xios::CContext::eventLoop(bool) ()
#42 0x00000000011ef4a8 in xios::CServerContext::threadEventLoop() ()
#43 0x00002ad520322814 in std::execute_native_thread_routine (__p=0x2ad66001b2d0)
at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#44 0x00002ad5222f7ea5 in start_thread () from /lib64/libpthread.so.0
#45 0x00002ad52290cb0d in clone () from /lib64/libc.so.6
comment:25 Changed 7 months ago by jderouillat
The integration of the communicator's tracking has been fixed (xios will no more look at what NetCDF or HDF5 are doing).
And so exceptions related to communicators management too, unless you're releasing the XIOS communicator by yourself (what we did in our sandbox ...).
As indicated elsewhere, the XIOS3 Head (2593 ) should be clean.
Note: See
TracTickets for help on using
tickets.
The chunking is defined per field, but with specifications done on the grid's elements to make easier the reuse of informations.
As you see, the chunking is based on :
Then :
The algorithm looks like to :
If no weight is defined, a chunking will be operated on the outside dimension to reach the target.