Changeset 2364 for branches/nemo_v3_3_beta
- Timestamp:
- 2010-11-05T16:22:12+01:00 (14 years ago)
- Location:
- branches/nemo_v3_3_beta
- Files:
-
- 3 added
- 32 edited
Legend:
- Unmodified
- Added
- Removed
-
branches/nemo_v3_3_beta/DOC/TexFiles/Chapters/Chap_MISC.tex
r2349 r2364 508 508 % Standard Model Output 509 509 % ------------------------------------------------------------------------------------------------------------- 510 \subsection{Model Output (default or \key{iomput} or \key{dimgout} )}510 \subsection{Model Output (default or \key{iomput} or \key{dimgout} or \key{netcdf4})} 511 511 \label{MISC_iom} 512 512 … … 536 536 an IEEE output format. 537 537 538 Since version 3.3, support for NetCDF4 chunking and (loss-less) compression has 539 been included. These options build on the standard NetCDF output and allow 540 the user control over the size of the chunks via namelist settings. Chunking 541 and compression can lead to significant reductions in file sizes for a small 542 runtime overhead. For a fuller discussion on chunking and other performance 543 issues the reader is referred to the NetCDF4 documentation: 544 http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html\#Chunking 545 546 The new features are only available when the code has been linked with a 547 NetCDF4 library (version 4.1 onwards, recommended) which has been built 548 with HDF5 support (version 1.8.4 onwards, recommended). Datasets created 549 with chunking and compression are not backwards compatible with NetCDF3 550 "classic" format but most analysis codes can be relinked simply with the 551 new libraries and will then read both NetCDF3 and NetCDF4 files. NEMO 552 executables linked with NetCDF4 libraries can be made to produce NetCDF3 553 files by setting the \np{ln\_nc4zip} logical to false in the \np{namnc4} 554 namelist: 555 556 %------------------------------------------namnc4---------------------------------------------------- 557 \namdisplay{namnc4} 558 %------------------------------------------------------------------------------------------------------------- 559 560 If \key{netcdf4} has not been defined, these namelist parameters are not read. 561 In this case, \np{ln\_nc4zip} is set false and dummy routines for a few 562 NetCDF4-specific functions are defined. These functions will not be used but 563 need to be included so that compilation is possible with NetCDF3 libraries. 564 565 When using NetCDF4 libraries, \key{netcdf4} should be defined even if the 566 intention is to create only NetCDF3-compatible files. This is necessary to 567 avoid duplication between the dummy routines and the actual routines present 568 in the library. Most compilers will fail at compile time when faced with 569 such duplication. Thus when linking with NetCDF4 libraries the user must 570 define \key{netcdf4} and control the type of NetCDF file produced via the 571 namelist parameter. 572 573 Chunking and compression is applied only to 4D fields and there is no 574 advantage in chunking across more than one time dimension since previously 575 written chunks would have to be read back and decompressed before being 576 added to. Therefore, user control over chunk sizes is provided only for the 577 three space dimensions. The user sets an approximate number of chunks along 578 each spatial axis. The actual size of the chunks will depend on global domain 579 size for mono-processors or, more likely, the local processor domain size for 580 distributed processing. The derived values are subject to practical minimum 581 values (to avoid wastefully small chunk sizes) and cannot be greater than the 582 domain size in any dimension. The algorithm used is: 583 584 \begin{alltt} {{\tiny 585 \begin{verbatim} 586 ichunksz(1) = MIN( idomain_size,MAX( (idomain_size-1)/nn_nchunks_i + 1 ,16 ) ) 587 ichunksz(2) = MIN( jdomain_size,MAX( (jdomain_size-1)/nn_nchunks_j + 1 ,16 ) ) 588 ichunksz(3) = MIN( kdomain_size,MAX( (kdomain_size-1)/nn_nchunks_k + 1 , 1 ) ) 589 ichunksz(4) = 1 590 \end{verbatim} 591 }}\end{alltt} 592 593 \noindent As an example, setting: 594 \vspace{-20pt} 595 \begin{alltt} {{\tiny 596 \begin{verbatim} 597 nn_nchunks_i=4, nn_nchunks_j=4 and nn_nchunks_k=31 598 \end{verbatim} 599 }}\end{alltt} \vspace{-10pt} 600 601 \noindent for a standard ORCA2\_LIM configuration gives chunksizes of {\small\tt 46x38x1} 602 respectively in the mono-processor case (i.e. global domain of {\small\tt 182x149x31}). 603 An illustration of the potential space savings that NetCDF4 chunking and compression 604 provides is given in table \ref{Tab_NC4} which compares the results of two short 605 runs of the ORCA2\_LIM reference configuration with a 4x2 mpi partitioning. Note 606 the variation in the compression ratio achieved which reflects chiefly the dry to wet 607 volume ratio of each processing region. 608 609 \begin{table} 610 \begin{tabular}{lrrr} 611 Filename & NetCDF3 & NetCDF4 & Reduction\\ 612 &filesize & filesize & \% \\ 613 &(KB) & (KB) & \\ 614 ORCA2\_restart\_0000.nc & 16420 & 8860 & 47\%\\ 615 ORCA2\_restart\_0001.nc & 16064 & 11456 & 29\%\\ 616 ORCA2\_restart\_0002.nc & 16064 & 9744 & 40\%\\ 617 ORCA2\_restart\_0003.nc & 16420 & 9404 & 43\%\\ 618 ORCA2\_restart\_0004.nc & 16200 & 5844 & 64\%\\ 619 ORCA2\_restart\_0005.nc & 15848 & 8172 & 49\%\\ 620 ORCA2\_restart\_0006.nc & 15848 & 8012 & 50\%\\ 621 ORCA2\_restart\_0007.nc & 16200 & 5148 & 69\%\\ 622 ORCA2\_2d\_grid\_T\_0000.nc & 2200 & 1504 & 32\%\\ 623 ORCA2\_2d\_grid\_T\_0001.nc & 2200 & 1748 & 21\%\\ 624 ORCA2\_2d\_grid\_T\_0002.nc & 2200 & 1592 & 28\%\\ 625 ORCA2\_2d\_grid\_T\_0003.nc & 2200 & 1540 & 30\%\\ 626 ORCA2\_2d\_grid\_T\_0004.nc & 2200 & 1204 & 46\%\\ 627 ORCA2\_2d\_grid\_T\_0005.nc & 2200 & 1444 & 35\%\\ 628 ORCA2\_2d\_grid\_T\_0006.nc & 2200 & 1428 & 36\%\\ 629 ORCA2\_2d\_grid\_T\_0007.nc & 2200 & 1148 & 48\%\\ 630 ... & ... & ... & .. \\ 631 ORCA2\_2d\_grid\_W\_0000.nc & 4416 & 2240 & 50\%\\ 632 ORCA2\_2d\_grid\_W\_0001.nc & 4416 & 2924 & 34\%\\ 633 ORCA2\_2d\_grid\_W\_0002.nc & 4416 & 2512 & 44\%\\ 634 ORCA2\_2d\_grid\_W\_0003.nc & 4416 & 2368 & 47\%\\ 635 ORCA2\_2d\_grid\_W\_0004.nc & 4416 & 1432 & 68\%\\ 636 ORCA2\_2d\_grid\_W\_0005.nc & 4416 & 1972 & 56\%\\ 637 ORCA2\_2d\_grid\_W\_0006.nc & 4416 & 2028 & 55\%\\ 638 ORCA2\_2d\_grid\_W\_0007.nc & 4416 & 1368 & 70\%\\ 639 \end{tabular} 640 \caption{\label{Tab_NC4} Filesize comparison between NetCDF3 and NetCDF4 641 with chunking and compression} 642 \end{table} 643 538 644 Since version 3.2, an I/O server has been added which provides more 539 645 flexibility in the choice of the fields to be output as well as how the 540 646 writing work is distributed over the processors in massively parallel 541 computing. It is activated when \key{iomput} is defined. 647 computing. It is activated when \key{iomput} is defined. 648 649 When \key{iomput} is activated with \key{netcdf4} chunking and 650 compression parameters for fields produced via \np{iom\_put} calls are 651 set via an equivalent and identically named namelist to \np{namnc4} in 652 \np{xmlio\_server.def}. Typically this namelist serves the mean files 653 whilst the \np{ namnc4} in the main namelist file continues to serve the 654 restart files. This duplication is unfortunate but appropriate since, if 655 using io\_servers, the domain sizes of the individual files produced by the 656 io\_server processes may be different to those produced by the invidual 657 processing regions and different chunking choices may be desired. 658 { 542 659 543 660 % ------------------------------------------------------------------------------------------------------------- -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/GYRE/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/GYRE/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 19 35 20 -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/GYRE_LOBSTER/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 19 35 20 -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/EXP00/1_namelist
r2360 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 19 35 20 -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM/cpp_ORCA2_LIM.fcm
r2340 r2364 1 bld::tool::fppkeys key_trabbl key_vectopt_loop key_orca_r2 key_lim2 key_dynspg_flt key_diaeiv key_ldfslp key_traldf_c2d key_traldf_eiv key_dynldf_c3d key_dtatem key_dtasal key_tradmp key_zdftke key_zdfddm key_iomput key_nproci=1 key_nprocj=11 bld::tool::fppkeys key_trabbl key_vectopt_loop key_orca_r2 key_lim2 key_dynspg_flt key_diaeiv key_ldfslp key_traldf_c2d key_traldf_eiv key_dynldf_c3d key_dtatem key_dtasal key_tradmp key_zdftke key_zdfddm key_nproci=4 key_nprocj=2 key_netcdf4 key_mpp_mpi -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_LIM_PISCES/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 19 35 20 -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/ORCA2_OFF_PISCES/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 35 -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/POMME/EXP00/namelist
r2340 r2364 42 42 cn_ocerst_in = "restart" ! suffix of ocean restart name (input) 43 43 cn_ocerst_out = "restart" ! suffix of ocean restart name (output) 44 / 45 !!====================================================================== 46 !! namnc4 netcdf4 chunking and compression settings 47 !!====================================================================== 48 !----------------------------------------------------------------------- 49 &namnc4 ! netcdf4 chunking and compression settings ("key_netcdf4") 50 ! (ignored if "key_netcdf4" is not used) 51 !----------------------------------------------------------------------- 52 nn_nchunks_i = 4 ! number of chunks in i-dimension 53 nn_nchunks_j = 4 ! number of chunks in j-dimension 54 nn_nchunks_k = 31 ! number of chunks in k-dimension 55 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 56 ! is optimal for postprocessing which works exclusively with horizontal slabs 57 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 58 ! (F) ignore chunking information and produce netcdf3-compatible files 44 59 / 45 60 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/CONFIG/POMME/EXP00/xmlio_server.def
r2281 r2364 17 17 global_mpi_buffer_size = 512 18 18 / 19 20 !!====================================================================== 21 !! namnc4 netcdf4 chunking and compression settings 22 !!====================================================================== 23 !----------------------------------------------------------------------- 24 &namnc4 ! netcdf4 chunking and compression settings 25 ! (benign if "key_netcdf4" is not used) 26 !----------------------------------------------------------------------- 27 nn_nchunks_i = 4 ! number of chunks in i-dimension 28 nn_nchunks_j = 4 ! number of chunks in j-dimension 29 nn_nchunks_k = 31 ! number of chunks in k-dimension 30 ! setting nn_nchunks_k = jpk will give a chunk size of 1 in the vertical which 31 ! is optimal for postprocessing which works exclusively with horizontal slabs 32 ln_nc4zip = .TRUE. ! (T) use netcdf4 chunking and compression 33 ! (F) ignore chunking information and produce netcdf3-compatible files 34 / 19 35 20 -
branches/nemo_v3_3_beta/NEMOGCM/EXTERNAL/IOIPSL/src/histcom.f90
r2281 r2364 7 7 !- 8 8 USE netcdf 9 USE nc4dummy ! needed to allow compilation with netcdf3 libraries 9 10 !- 10 11 USE stringop, ONLY : nocomma,cmpblank,findpos,find_str,strlowercase … … 35 36 !- to describe the grid, just two vectors. 36 37 !--------------------------------------------------------------------- 38 !- 39 TYPE, PUBLIC :: snc4_ctl !: netcdf4 chunking control structure 40 !: (optional on histbeg and histend calls) 41 SEQUENCE 42 INTEGER :: ni 43 INTEGER :: nj 44 INTEGER :: nk 45 LOGICAL :: luse 46 END TYPE snc4_ctl 37 47 !- 38 48 INTERFACE histbeg … … 163 173 & (pfilename,pim,plon,pjm,plat, & 164 174 & par_orix,par_szx,par_oriy,par_szy, & 165 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode )175 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode,snc4chunks) 166 176 !--------------------------------------------------------------------- 167 177 !- histbeg for 1D regular horizontal coordinates (see histb_all) … … 179 189 INTEGER,INTENT(IN),OPTIONAL :: domain_id 180 190 CHARACTER(LEN=*),OPTIONAL,INTENT(IN) :: mode 191 TYPE(snc4_ctl), OPTIONAL,INTENT(IN) :: snc4chunks 181 192 !--------------------------------------------------------------------- 182 193 CALL histb_all & … … 184 195 & x_1d=plon,y_1d=plat, & 185 196 & k_orx=par_orix,k_szx=par_szx,k_ory=par_oriy,k_szy=par_szy, & 186 & domain_id=domain_id,mode=mode )197 & domain_id=domain_id,mode=mode,snc4chunks=snc4chunks) 187 198 !------------------------- 188 199 END SUBROUTINE histb_reg1d … … 191 202 & (pfilename,pim,plon,pjm,plat, & 192 203 & par_orix,par_szx,par_oriy,par_szy, & 193 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode )204 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode,snc4chunks) 194 205 !--------------------------------------------------------------------- 195 206 !- histbeg for 2D regular horizontal coordinates (see histb_all) … … 206 217 INTEGER,INTENT(IN),OPTIONAL :: domain_id 207 218 CHARACTER(LEN=*),OPTIONAL,INTENT(IN) :: mode 219 TYPE(snc4_ctl), OPTIONAL,INTENT(IN) :: snc4chunks 208 220 !--------------------------------------------------------------------- 209 221 CALL histb_all & … … 211 223 & x_2d=plon,y_2d=plat, & 212 224 & k_orx=par_orix,k_szx=par_szx,k_ory=par_oriy,k_szy=par_szy, & 213 & domain_id=domain_id,mode=mode )225 & domain_id=domain_id,mode=mode,snc4chunks=snc4chunks) 214 226 !------------------------- 215 227 END SUBROUTINE histb_reg2d … … 217 229 SUBROUTINE histb_irreg & 218 230 & (pfilename,pim,plon,plon_bounds,plat,plat_bounds, & 219 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode )231 & pitau0,pdate0,pdeltat,phoriid,idf,domain_id,mode,snc4chunks) 220 232 !--------------------------------------------------------------------- 221 233 !- histbeg for irregular horizontal coordinates (see histb_all) … … 232 244 INTEGER,INTENT(IN),OPTIONAL :: domain_id 233 245 CHARACTER(LEN=*),OPTIONAL,INTENT(IN) :: mode 246 TYPE(snc4_ctl), OPTIONAL,INTENT(IN) :: snc4chunks 234 247 !--------------------------------------------------------------------- 235 248 CALL histb_all & 236 249 & (3,pfilename,pim,pim,pitau0,pdate0,pdeltat,phoriid,idf, & 237 250 & x_1d=plon,y_1d=plat,x_bnds=plon_bounds,y_bnds=plat_bounds, & 238 & domain_id=domain_id,mode=mode )251 & domain_id=domain_id,mode=mode,snc4chunks=snc4chunks) 239 252 !------------------------- 240 253 END SUBROUTINE histb_irreg … … 243 256 & (k_typ,nc_name,pim,pjm,pitau0,pdate0,pdeltat,phoriid,idf, & 244 257 & x_1d,y_1d,x_2d,y_2d,k_orx,k_szx,k_ory,k_szy, & 245 & x_bnds,y_bnds,domain_id,mode )258 & x_bnds,y_bnds,domain_id,mode,snc4chunks) 246 259 !--------------------------------------------------------------------- 247 260 !- General interface for horizontal grids. … … 305 318 !- Keywords "NETCDF4" and "CLASSIC" are reserved 306 319 !- for future use. 320 !- 321 !- snc4chunks : Structure containing chunk partitioning parameters 322 !- for 4-D variables and a logical switch to toggle 323 !- between netcdf3 o/p (false) and netcdf4 chunked 324 !- and compressed o/p (true) 307 325 !--------------------------------------------------------------------- 308 326 IMPLICIT NONE … … 320 338 INTEGER,INTENT(IN),OPTIONAL :: domain_id 321 339 CHARACTER(LEN=*),OPTIONAL,INTENT(IN) :: mode 340 TYPE(snc4_ctl), OPTIONAL,INTENT(IN) :: snc4chunks 322 341 !- 323 342 INTEGER :: nfid,iret,m_c … … 406 425 ELSE 407 426 m_c = IOR(NF90_CLOBBER,NF90_64BIT_OFFSET) 427 ENDIF 428 !- 429 IF (PRESENT(snc4chunks)) THEN 430 IF (snc4chunks%luse) m_c = NF90_HDF5 ! will conflict if compiling with netcdf4 libraries without "key_netcdf4" 408 431 ENDIF 409 432 !- … … 1347 1370 END SUBROUTINE histdef 1348 1371 !=== 1349 SUBROUTINE histend (idf )1372 SUBROUTINE histend (idf, snc4chunks) 1350 1373 !--------------------------------------------------------------------- 1351 1374 !- This subroutine end the decalaration of variables and sets the … … 1362 1385 !- 1363 1386 INTEGER,INTENT(IN) :: idf 1387 TYPE(snc4_ctl), OPTIONAL,INTENT(IN) :: snc4chunks 1364 1388 !- 1365 1389 INTEGER :: nfid,nvid,iret,ndim,iv,itx,ziv,itax,dim_cnt … … 1378 1402 LOGICAL :: l_b 1379 1403 LOGICAL :: l_dbg 1404 INTEGER, DIMENSION(4) :: ichunksz ! NETCDF4 chunk sizes 1405 INTEGER :: ichunkalg, ishuffle,& 1406 ideflate, ideflate_level 1407 LOGICAL :: lchunk = .FALSE. ! logical switch to activate chunking when appropriate 1408 !- 1409 ! NetCDF4 chunking and compression parameters 1410 ichunkalg = 0 1411 ishuffle = 1 1412 ideflate = 1 1413 ideflate_level = 1 1414 ! 1380 1415 !--------------------------------------------------------------------- 1381 1416 CALL ipsldbg (old_status=l_dbg) … … 1517 1552 & W_F(idf)%W_V(iv)%v_typ,dims(1:ABS(ndim)),nvid) 1518 1553 !- 1554 IF( ndim == 4 ) THEN 1555 IF( PRESENT( snc4chunks ) ) THEN 1556 IF( snc4chunks%luse ) THEN 1557 ichunksz = 1 1558 iret = NF90_INQUIRE_DIMENSION( nfid, W_F(idf)%xid, len = ichunksz(1) ) 1559 iret = NF90_INQUIRE_DIMENSION( nfid, W_F(idf)%yid, len = ichunksz(2) ) 1560 IF ( ziv .NE. -99 ) & 1561 iret = NF90_INQUIRE_DIMENSION( nfid, W_F(idf)%zax_ids(ziv), len = ichunksz(3) ) 1562 ichunksz(1) = MIN(ichunksz(1), MAX((ichunksz(1)-1)/snc4chunks%ni + 1,16)) 1563 ichunksz(2) = MIN(ichunksz(2), MAX((ichunksz(2)-1)/snc4chunks%nj + 1,16)) 1564 ichunksz(3) = MIN(ichunksz(3), MAX((ichunksz(3)-1)/snc4chunks%nk + 1, 1)) 1565 ! Always use a chunk size of 1 for the unlimited dimension 1566 iret = NF90_DEF_VAR_CHUNKING(nfid, nvid, ichunkalg, ichunksz) ! will conflict if compiling with netcdf4 libraries without "key_netcdf4" 1567 1568 iret = NF90_DEF_VAR_DEFLATE(nfid, nvid, ishuffle, ideflate, ideflate_level) ! will conflict if compiling with netcdf4 libraries without "key_netcdf4" 1569 1570 ENDIF 1571 ENDIF 1572 ENDIF 1519 1573 W_F(idf)%W_V(iv)%ncvid = nvid 1520 1574 !- -
branches/nemo_v3_3_beta/NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_interface_ioipsl.f90
r2281 r2364 38 38 USE ioipsl 39 39 USE xmlio 40 USE mod_ioserver_namelist 40 41 IMPLICIT NONE 41 42 INTEGER,INTENT(IN) :: nb_server … … 85 86 pt_zoom%ibegin_loc, pt_zoom%ni_loc,pt_zoom%jbegin_loc,pt_zoom%nj_loc, & 86 87 initial_timestep, initial_date, timestep_value, & 87 ioipsl_hori_id, ioipsl_file_id )88 ioipsl_hori_id, ioipsl_file_id, snc4chunks=snc4ioset) 88 89 ELSE 89 90 … … 92 93 pt_zoom%ibegin_loc, pt_zoom%ni_loc,pt_zoom%jbegin_loc,pt_zoom%nj_loc, & 93 94 initial_timestep, initial_date, timestep_value, & 94 ioipsl_hori_id, ioipsl_file_id,domain_id=ioipsl_domain_id )95 ioipsl_hori_id, ioipsl_file_id,domain_id=ioipsl_domain_id, snc4chunks=snc4ioset) 95 96 96 97 ENDIF … … 139 140 ENDIF 140 141 ENDDO 141 CALL histend(ioipsl_file_id )142 CALL histend(ioipsl_file_id, snc4chunks=snc4ioset) 142 143 ENDIF 143 144 CALL sorted_list__delete(axis_id) -
branches/nemo_v3_3_beta/NEMOGCM/EXTERNAL/XMLIO_SERVER/src/IOSERVER/mod_ioserver_namelist.f90
r2281 r2364 11 11 12 12 INTEGER,PARAMETER,PRIVATE :: starting_unit=356 13 TYPE, PUBLIC :: snc4_ctl !: netcdf4 chunking control structure 14 !: (optional on histbeg and histend calls) 15 SEQUENCE 16 INTEGER :: ni 17 INTEGER :: nj 18 INTEGER :: nk 19 LOGICAL :: luse 20 END TYPE snc4_ctl 21 TYPE(snc4_ctl), SAVE :: snc4ioset 22 INTEGER :: nn_nchunks_i, nn_nchunks_j, nn_nchunks_k 23 LOGICAL :: ln_nc4zip=.false. 24 NAMELIST /namnc4/ nn_nchunks_i, nn_nchunks_j, nn_nchunks_k, ln_nc4zip 13 25 14 26 CONTAINS … … 22 34 server_id='unknown' 23 35 global_mpi_buffer_size=512 36 snc4ioset%luse = .false. 24 37 25 38 END SUBROUTINE set_default_namelist … … 46 59 READ(unit,nml=coupling_param) 47 60 READ(unit,nml=mpi_param) 48 global_mpi_buffer_size=global_mpi_buffer_size*1024*128 61 READ(unit,nml=namnc4,ERR=666,END=666) 62 666 global_mpi_buffer_size=global_mpi_buffer_size*1024*128 49 63 CLOSE(unit) 64 IF (ln_nc4zip) THEN 65 snc4ioset%ni = nn_nchunks_i 66 snc4ioset%nj = nn_nchunks_j 67 snc4ioset%nk = nn_nchunks_k 68 snc4ioset%luse = ln_nc4zip 69 ENDIF 50 70 ELSE 51 71 PRINT *,'WARNING : mod_ioserver::read_namelist : file xmlio_server.def is absent', & -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/C1D_SRC/diawri_c1d.F90
r2303 r2364 142 142 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & ! Horizontal grid: glamt and gphit 143 143 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 144 & nit000-1, zjulian, zdt, nh_T, nid_T, domain_id=nidom )144 & nit000-1, zjulian, zdt, nh_T, nid_T, domain_id=nidom, snc4chunks=snc4set ) 145 145 CALL histvert( nid_T, "deptht", "Vertical T levels", & ! Vertical grid: gdept 146 146 & "m", ipk, gdept_0, nz_T, "down" ) … … 243 243 ENDIF 244 244 245 CALL histend( nid_T )245 CALL histend( nid_T, snc4set ) 246 246 247 247 IF(lwp) WRITE(numout,*) -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/LIM_SRC_2/limwri_2.F90
r2287 r2364 112 112 CALL dia_nam ( clhstnam, nwrite, 'icemod' ) 113 113 CALL histbeg ( clhstnam, jpi, glamt, jpj, gphit, & 114 & 1, jpi, 1, jpj, niter, zjulian, rdt_ice, nhorid, nice , domain_id=nidom )114 & 1, jpi, 1, jpj, niter, zjulian, rdt_ice, nhorid, nice , domain_id=nidom, snc4chunks=snc4set) 115 115 CALL histvert( nice, "deptht", "Vertical T levels", "m", 1, zdept, ndepid, "down") 116 116 CALL wheneq ( jpij , tmask(:,:,1), 1, 1., ndex51, ndim) … … 120 120 & , nhorid, 1, 1, 1, -99, 32, clop, zsto, zout ) 121 121 END DO 122 CALL histend( nice )122 CALL histend( nice, snc4set ) 123 123 ! 124 124 ENDIF … … 305 305 CALL histdef( kid, "isnowpre", "Snow precipitation" , "kg/m2/s", jpi, jpj, kh_i, 1, 1, 1, -99, 32, "inst(x)", rdt, rdt ) 306 306 307 CALL histend( kid ) ! end of the file definition307 CALL histend( kid, snc4set ) ! end of the file definition 308 308 309 309 CALL histwrite( kid, "isnowthi", kt, hsnif , jpi*jpj, (/1/) ) -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/LIM_SRC_3/limwri.F90
r2287 r2364 146 146 CALL dia_nam ( clhstnam, nwrite, 'icemod' ) 147 147 CALL histbeg ( clhstnam, jpi, glamt, jpj, gphit, 1, jpi, 1, jpj, niter, zjulian, rdt_ice, & 148 & nhorid, nice, domain_id=nidom )148 & nhorid, nice, domain_id=nidom, snc4chunks=snc4set ) 149 149 CALL histvert( nice, "deptht", "Vertical T levels", "m", 1, zdept, ndepid, "down") 150 150 CALL wheneq ( jpij , tmask(:,:,1), 1, 1., ndex51, ndim) … … 160 160 END DO 161 161 162 CALL histend(nice )162 CALL histend(nice, snc4set) 163 163 164 164 !----------------- … … 175 175 niter, zjulian, rdt_ice, & ! time 176 176 nhorida, & ! ? linked with horizontal ... 177 nicea , domain_id=nidom ) ! file177 nicea , domain_id=nidom, snc4chunks=snc4set) ! file 178 178 CALL histvert( nicea, "icethi", "L levels", & 179 179 "m", ipl , hi_mean , nz ) … … 195 195 CALL histdef( nicea, "iice_etd", "Brine volume distr. " , "%" , & 196 196 jpi, jpj, nhorida, jpl, 1, jpl, nz, 15, clop, zsto, zout ) 197 CALL histend(nicea )197 CALL histend(nicea, snc4set) 198 198 ENDIF 199 199 -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/DIA/diaptr.F90
r2287 r2364 728 728 ! Horizontal grid : zphi() 729 729 CALL histbeg(clhstnam, 1, zfoo, jpj, zphi, & 730 1, 1, 1, jpj, niter, zjulian, zdt*nf_ptr, nhoridz, numptr, domain_id=nidom_ptr )730 1, 1, 1, jpj, niter, zjulian, zdt*nf_ptr, nhoridz, numptr, domain_id=nidom_ptr, snc4chunks=snc4set) 731 731 ! Vertical grids : gdept_0, gdepw_0 732 732 CALL histvert( numptr, "deptht", "Vertical T levels", & … … 880 880 ENDIF 881 881 882 CALL histend( numptr )882 CALL histend( numptr, snc4set ) 883 883 884 884 END IF -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/DIA/diawri.F90
r2329 r2364 273 273 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & ! Horizontal grid: glamt and gphit 274 274 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 275 & nit000-1, zjulian, zdt, nh_T, nid_T, domain_id=nidom )275 & nit000-1, zjulian, zdt, nh_T, nid_T, domain_id=nidom, snc4chunks=snc4set ) 276 276 CALL histvert( nid_T, "deptht", "Vertical T levels", & ! Vertical grid: gdept 277 277 & "m", ipk, gdept_0, nz_T, "down" ) … … 286 286 CALL histbeg( clhstnam, jpi, glamu, jpj, gphiu, & ! Horizontal grid: glamu and gphiu 287 287 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 288 & nit000-1, zjulian, zdt, nh_U, nid_U, domain_id=nidom )288 & nit000-1, zjulian, zdt, nh_U, nid_U, domain_id=nidom, snc4chunks=snc4set ) 289 289 CALL histvert( nid_U, "depthu", "Vertical U levels", & ! Vertical grid: gdept 290 290 & "m", ipk, gdept_0, nz_U, "down" ) … … 299 299 CALL histbeg( clhstnam, jpi, glamv, jpj, gphiv, & ! Horizontal grid: glamv and gphiv 300 300 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 301 & nit000-1, zjulian, zdt, nh_V, nid_V, domain_id=nidom )301 & nit000-1, zjulian, zdt, nh_V, nid_V, domain_id=nidom, snc4chunks=snc4set ) 302 302 CALL histvert( nid_V, "depthv", "Vertical V levels", & ! Vertical grid : gdept 303 303 & "m", ipk, gdept_0, nz_V, "down" ) … … 312 312 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & ! Horizontal grid: glamt and gphit 313 313 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 314 & nit000-1, zjulian, zdt, nh_W, nid_W, domain_id=nidom )314 & nit000-1, zjulian, zdt, nh_W, nid_W, domain_id=nidom, snc4chunks=snc4set ) 315 315 CALL histvert( nid_W, "depthw", "Vertical W levels", & ! Vertical grid: gdepw 316 316 & "m", ipk, gdepw_0, nz_W, "down" ) … … 404 404 #endif 405 405 406 CALL histend( nid_T )406 CALL histend( nid_T, snc4chunks=snc4set ) 407 407 408 408 ! !!! nid_U : 3D … … 417 417 & jpi, jpj, nh_U, 1 , 1, 1 , - 99, 32, clop, zsto, zout ) 418 418 419 CALL histend( nid_U )419 CALL histend( nid_U, snc4chunks=snc4set ) 420 420 421 421 ! !!! nid_V : 3D … … 430 430 & jpi, jpj, nh_V, 1 , 1, 1 , - 99, 32, clop, zsto, zout ) 431 431 432 CALL histend( nid_V )432 CALL histend( nid_V, snc4chunks=snc4set ) 433 433 434 434 ! !!! nid_W : 3D … … 458 458 #endif 459 459 460 CALL histend( nid_W )460 CALL histend( nid_W, snc4chunks=snc4set ) 461 461 462 462 IF(lwp) WRITE(numout,*) … … 624 624 zjulian = zjulian - adatrj ! set calendar origin to the beginning of the experiment 625 625 CALL histbeg( clname, jpi, glamt, jpj, gphit, & 626 1, jpi, 1, jpj, nit000-1, zjulian, zdt, nh_i, id_i, domain_id=nidom )! Horizontal grid : glamt and gphit626 1, jpi, 1, jpj, nit000-1, zjulian, zdt, nh_i, id_i, domain_id=nidom, snc4chunks=snc4set ) ! Horizontal grid : glamt and gphit 627 627 CALL histvert( id_i, "deptht", "Vertical T levels", & ! Vertical grid : gdept 628 628 "m", jpk, gdept_0, nz_i, "down") … … 658 658 CALL lim_wri_state_2( kt, id_i, nh_i ) 659 659 #else 660 CALL histend( id_i )660 CALL histend( id_i, snc4chunks=snc4set ) 661 661 #endif 662 662 -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/DOM/domain.F90
r2287 r2364 106 106 !! - namdom namelist 107 107 !! - namcla namelist 108 !! - namnc4 namelist ! "key_netcdf4" only 108 109 !!---------------------------------------------------------------------- 109 110 USE ioipsl … … 115 116 & rn_rdtmax, rn_rdth , nn_baro , nn_closea 116 117 NAMELIST/namcla/ nn_cla 118 #if defined key_netcdf4 119 NAMELIST/namnc4/ nn_nchunks_i, nn_nchunks_j, nn_nchunks_k, ln_nc4zip 120 #endif 117 121 !!---------------------------------------------------------------------- 118 122 … … 232 236 IF( lk_mpp_rep .AND. n_cla /= 0 ) CALL ctl_stop( ' Reproductibility tests (nbit_cmp=1) require n_cla = 0' ) 233 237 ! 238 239 #if defined key_netcdf4 240 241 REWIND( numnam ) ! Namelist namnc4 : netcdf4 chunking parameters 242 READ ( numnam, namnc4 ) 243 IF(lwp) THEN 244 WRITE(numout,*) 245 WRITE(numout,*) ' Namelist namnc4 - Netcdf4 chunking parameters' 246 WRITE(numout,*) ' number of chunks in i-dimension nn_nchunks_i = ', nn_nchunks_i 247 WRITE(numout,*) ' number of chunks in j-dimension nn_nchunks_j = ', nn_nchunks_j 248 WRITE(numout,*) ' number of chunks in k-dimension nn_nchunks_k = ', nn_nchunks_k 249 WRITE(numout,*) ' apply netcdf4/hdf5 chunking & compression ln_nc4zip = ', ln_nc4zip 250 ENDIF 251 252 ! Put the netcdf4 settings into a simple structure (snc4set, defined in in_out_manager module) 253 ! Note the chunk size in the unlimited (time) dimension will be fixed at 1 254 snc4set%ni = nn_nchunks_i 255 snc4set%nj = nn_nchunks_j 256 snc4set%nk = nn_nchunks_k 257 snc4set%luse = ln_nc4zip 258 259 #else 260 261 snc4set%luse = .FALSE. 262 263 #endif 234 264 END SUBROUTINE dom_nam 235 265 -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/IOM/in_out_manager.F90
r2287 r2364 45 45 LOGICAL :: ln_clobber = .FALSE. !: clobber (overwrite) an existing file 46 46 INTEGER :: nn_chunksz = 0 !: chunksize (bytes) for NetCDF file (working only with iom_nf90 routines) 47 #if defined key_netcdf4 48 !!---------------------------------------------------------------------- 49 !! namnc4 namelist parameters 50 !!---------------------------------------------------------------------- 51 !: ========================================================================= 52 !: The following four values determine the partitioning of the output fields 53 !: into netcdf4 chunks. They are unrelated to the nn_chunk_sz setting which is 54 !: for runtime optimisation. The individual netcdf4 chunks can be optionally 55 !: gzipped (recommended) leading to significant reductions in I/O volumes 56 INTEGER :: nn_nchunks_i = 1 !: number of chunks required in the i-dimension (only with iom_nf90 routines and key_netcdf4) 57 INTEGER :: nn_nchunks_j = 1 !: number of chunks required in the j-dimension (only with iom_nf90 routines and key_netcdf4) 58 INTEGER :: nn_nchunks_k = 1 !: number of chunks required in the k-dimension (only with iom_nf90 routines and key_netcdf4) 59 INTEGER :: nn_nchunks_t = 1 !: number of chunks required in the t-dimension (only with iom_nf90 routines and key_netcdf4) 60 LOGICAL :: ln_nc4zip = .TRUE. !: netcdf4 usage. (T): chunk and compress output datasets using the HDF5 sublayers of netcdf4 61 !: (F): ignore chunking request and use the netcdf4 library to produce netcdf3-compatible files 62 #endif 63 64 !$AGRIF_DO_NOT_TREAT 65 TYPE, PUBLIC :: snc4_ctl !: netcdf4 chunking control structure (always needed for decision making) 66 SEQUENCE 67 INTEGER :: ni 68 INTEGER :: nj 69 INTEGER :: nk 70 LOGICAL :: luse 71 END TYPE snc4_ctl 72 73 TYPE(snc4_ctl) :: snc4set !: netcdf4 chunking control structure (always needed for decision making) 74 !$AGRIF_END_DO_NOT_TREAT 75 47 76 48 77 !! conversion of DOCTOR norm namelist name into model name -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/IOM/iom_nf90.F90
r2351 r2364 39 39 MODULE PROCEDURE iom_nf90_rp0123d 40 40 END INTERFACE 41 42 #if ! defined key_netcdf4 43 !!-------------------------------------------------------------------- 44 !! NOT 'key_netcdf4' Defines dummy routines for netcdf4 45 !! calls when compiling without netcdf4 libraries 46 !!-------------------------------------------------------------------- 47 PUBLIC NF90_DEF_VAR_CHUNKING, NF90_DEF_VAR_DEFLATE ! contained below 48 INTEGER :: NF90_HDF5 49 #endif 41 50 !!---------------------------------------------------------------------- 42 51 !! NEMO/OPA 3.3 , NEMO Consortium (2010) … … 67 76 INTEGER :: jl ! loop variable 68 77 INTEGER :: ichunk ! temporary storage of nn_chunksz 69 INTEGER :: imode ! creation mode flag: NF90_CLOBBER or NF90_NOCLOBBER 78 INTEGER :: imode ! creation mode flag: NF90_CLOBBER or NF90_NOCLOBBER or NF90_HDF5 70 79 LOGICAL :: llclobber ! local definition of ln_clobber 71 80 !--------------------------------------------------------------------- … … 82 91 IF( ldwrt ) THEN ! ... in write mode 83 92 IF(lwp) WRITE(numout,*) TRIM(clinfo)//' open existing file: '//TRIM(cdname)//' in WRITE mode' 84 CALL iom_nf90_check(NF90_OPEN( TRIM(cdname), NF90_WRITE , if90id, chunksize = ichunk ), clinfo) 93 IF( snc4set%luse ) THEN 94 CALL iom_nf90_check(NF90_OPEN( TRIM(cdname), NF90_WRITE , if90id ), clinfo) 95 ELSE 96 CALL iom_nf90_check(NF90_OPEN( TRIM(cdname), NF90_WRITE , if90id, chunksize = ichunk ), clinfo) 97 ENDIF 85 98 CALL iom_nf90_check(NF90_SET_FILL( if90id, NF90_NOFILL, idmy ), clinfo) 86 99 ELSE ! ... in read mode … … 101 114 ELSE ; imode = IOR( NF90_64BIT_OFFSET, NF90_NOCLOBBER ) 102 115 ENDIF 103 CALL iom_nf90_check(NF90_CREATE( TRIM(cdname), imode, if90id, chunksize = ichunk ), clinfo) 116 IF( snc4set%luse ) THEN 117 IF(lwp) WRITE(numout,*) TRIM(clinfo)//' creating file: '//TRIM(cdname)//' in hdf5 (netcdf4) mode' 118 IF( llclobber ) THEN ; imode = IOR(NF90_HDF5, NF90_CLOBBER) 119 ELSE ; imode = IOR(NF90_HDF5, NF90_NOCLOBBER) 120 ENDIF 121 CALL iom_nf90_check(NF90_CREATE( TRIM(cdname), imode, if90id ), clinfo) 122 ELSE 123 CALL iom_nf90_check(NF90_CREATE( TRIM(cdname), imode, if90id, chunksize = ichunk ), clinfo) 124 ENDIF 104 125 CALL iom_nf90_check(NF90_SET_FILL( if90id, NF90_NOFILL, idmy ), clinfo) 105 126 ! define dimensions … … 378 399 INTEGER :: idmy ! dummy variable 379 400 INTEGER :: itype ! variable type 401 INTEGER, DIMENSION(4) :: ichunksz ! NetCDF4 chunk sizes. Will be computed using 402 ! nn_nchunks_[i,j,k,t] namelist parameters 403 INTEGER :: ichunkalg, ishuffle,& 404 ideflate, ideflate_level 405 ! NetCDF4 internally fixed parameters 406 LOGICAL :: lchunk ! logical switch to activate chunking and compression 407 ! when appropriate (currently chunking is applied to 4d fields only) 380 408 !--------------------------------------------------------------------- 381 409 ! … … 408 436 ! =============== 409 437 IF( kvid <= 0 ) THEN 438 ! 439 ! NetCDF4 chunking and compression fixed settings 440 ichunkalg = 0 441 ishuffle = 1 442 ideflate = 1 443 ideflate_level = 1 444 ! 410 445 idvar = iom_file(kiomid)%nvars + 1 411 446 ! are we in define mode? … … 438 473 & iom_file(kiomid)%nvid(idvar) ), clinfo) 439 474 ENDIF 475 lchunk = .false. 476 IF( snc4set%luse .AND. idims.eq.4 ) lchunk = .true. 440 477 ! update informations structure related the new variable we want to add... 441 478 iom_file(kiomid)%nvars = idvar … … 449 486 DO jd = 1, idims 450 487 CALL iom_nf90_check(NF90_INQUIRE_DIMENSION( if90id, idimid(jd), len = iom_file(kiomid)%dimsz(jd,idvar) ), clinfo) 488 IF ( lchunk ) ichunksz(jd) = iom_file(kiomid)%dimsz(jd,idvar) 451 489 END DO 490 IF ( lchunk ) THEN 491 ! Calculate chunk sizes by partitioning each dimension as requested in namnc4 namelist 492 ! Disallow very small chunk sizes and prevent chunk sizes larger than each individual dimension 493 ichunksz(1) = MIN( ichunksz(1),MAX( (ichunksz(1)-1)/snc4set%ni + 1 ,16 ) ) ! Suggested default nc4set%ni=4 494 ichunksz(2) = MIN( ichunksz(2),MAX( (ichunksz(2)-1)/snc4set%nj + 1 ,16 ) ) ! Suggested default nc4set%nj=2 495 ichunksz(3) = MIN( ichunksz(3),MAX( (ichunksz(3)-1)/snc4set%nk + 1 , 1 ) ) ! Suggested default nc4set%nk=6 496 ichunksz(4) = 1 ! Do not allow chunks to span the 497 ! unlimited dimension 498 CALL iom_nf90_check(NF90_DEF_VAR_CHUNKING(if90id, idvar, ichunkalg, ichunksz), clinfo) 499 CALL iom_nf90_check(NF90_DEF_VAR_DEFLATE(if90id, idvar, ishuffle, ideflate, ideflate_level), clinfo) 500 IF(lwp) WRITE(numout,*) TRIM(clinfo)//' chunked ok. Chunks sizes: ', ichunksz 501 ENDIF 452 502 IF(lwp) WRITE(numout,*) TRIM(clinfo)//' defined ok' 453 503 ELSE … … 529 579 END SUBROUTINE iom_nf90_check 530 580 581 #if ! defined key_netcdf4 582 583 !!-------------------------------------------------------------------- 584 !! NOT 'key_netcdf4' Defines dummy routines for netcdf4 585 !! calls when compiling without netcdf4 libraries 586 !!-------------------------------------------------------------------- 587 588 INTEGER FUNCTION NF90_DEF_VAR_CHUNKING(idum1, idum2, idum3, iarr1) 589 !!-------------------------------------------------------------------- 590 !! *** SUBROUTINE NF90_DEF_VAR_CHUNKING *** 591 !! 592 !! ** Purpose : Dummy NetCDF4 routine to enable compiling with NetCDF3 libraries 593 !!-------------------------------------------------------------------- 594 INTEGER, INTENT(in) :: idum1, idum2, idum3 595 INTEGER, DIMENSION(4), INTENT(in) :: iarr1 596 IF(lwp) WRITE(numout,*) 'Warning: Attempt to chunk output variable without NetCDF4 support' 597 NF90_DEF_VAR_CHUNKING = -1 598 END FUNCTION NF90_DEF_VAR_CHUNKING 599 600 INTEGER FUNCTION NF90_DEF_VAR_DEFLATE(idum1, idum2, idum3, idum4, idum5) 601 !!-------------------------------------------------------------------- 602 !! *** SUBROUTINE NF90_DEF_VAR_DEFLATE *** 603 !! 604 !! ** Purpose : Dummy NetCDF4 routine to enable compiling with NetCDF3 libraries 605 !!-------------------------------------------------------------------- 606 INTEGER, INTENT(in) :: idum1, idum2, idum3, idum4, idum5 607 IF(lwp) WRITE(numout,*) 'Warning: Attempt to compress output variable without NetCDF4 support' 608 NF90_DEF_VAR_DEFLATE = -1 609 END FUNCTION NF90_DEF_VAR_DEFLATE 610 #endif 531 611 532 612 !!====================================================================== -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/IOM/restart.F90
r2287 r2364 95 95 CASE DEFAULT ; WRITE(numout,*) ' open ocean restart NetCDF file: '//clname 96 96 END SELECT 97 IF ( snc4set%luse ) WRITE(numout,*) ' opened for NetCDF4 chunking and compression' 97 98 IF( kt == nitrst - 1 ) THEN ; WRITE(numout,*) ' kt = nitrst - 1 = ', kt 98 99 ELSE ; WRITE(numout,*) ' kt = ' , kt … … 178 179 CASE ( jprstdimg ) ; WRITE(numout,*) 'rst_read : read oce binary restart file' 179 180 END SELECT 181 IF ( snc4set%luse ) WRITE(numout,*) 'rst_read : configured with NetCDF4 support' 180 182 WRITE(numout,*) '~~~~~~~~' 181 183 ENDIF -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/TRD/trdmld.F90
r2287 r2364 873 873 IF(lwp) WRITE(numout,*) ' Name of NETCDF file ', clhstnam 874 874 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 875 & 1, jpi, 1, jpj, nit000-1, zjulian, rdt, nh_t, nidtrd, domain_id=nidom )875 & 1, jpi, 1, jpj, nit000-1, zjulian, rdt, nh_t, nidtrd, domain_id=nidom, snc4chunks=snc4set ) 876 876 877 877 !-- Define the ML depth variable … … 925 925 926 926 !-- Leave IOIPSL/NetCDF define mode 927 CALL histend( nidtrd )927 CALL histend( nidtrd, snc4set ) 928 928 929 929 #endif /* key_dimgout */ -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/OPA_SRC/TRD/trdvor.F90
r2287 r2364 568 568 IF(lwp) WRITE(numout,*) ' Name of NETCDF file ', clhstnam 569 569 CALL histbeg( clhstnam, jpi, glamf, jpj, gphif,1, jpi, & ! Horizontal grid : glamt and gphit 570 & 1, jpj, nit000-1, zjulian, rdt, nh_t, nidvor, domain_id=nidom )570 & 1, jpj, nit000-1, zjulian, rdt, nh_t, nidvor, domain_id=nidom, snc4chunks=snc4set ) 571 571 CALL wheneq( jpi*jpj, fmask, 1, 1., ndexvor1, ndimvor1 ) ! surface 572 572 … … 598 598 CALL histdef( nidvor, "sovorgap", cvort//"gap", "s-2", & ! gap between 1st and 2 nd mbre 599 599 & jpi,jpj,nh_t,1,1,1,-99,32,clop,zsto,zout) 600 CALL histend( nidvor )600 CALL histend( nidvor, snc4set ) 601 601 602 602 IF( idebug /= 0 ) THEN -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/TOP_SRC/SED/sedwri.F90
r2281 r2364 186 186 CALL histbeg ( clhstnam, jpi, glamt, jpj, gphit, & 187 187 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 188 & nitsed000-1, zjulian, zdt, nhorised, nised , domain_id=nidom )188 & nitsed000-1, zjulian, zdt, nhorised, nised , domain_id=nidom, snc4chunks=snc4set ) 189 189 CALL histvert( nised,'deptht','Vertic.sed.T levels','m',ipk, profsed, ndepsed, 'down' ) 190 190 CALL wheneq ( jpi*jpj*ipk, tmasksed, 1, 1., ndext52, ndimt52 ) … … 223 223 224 224 225 CALL histend( nised )225 CALL histend( nised, snc4set ) 226 226 227 227 WRITE(numsed,*) -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/TOP_SRC/TRP/trdmld_trc.F90
r2287 r2364 1304 1304 CALL dia_nam( clhstnam, nn_trd_trc, csuff ) 1305 1305 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 1306 & 1, jpi, 1, jpj, nit000, zjulian, rdt, nh_t(jn), nidtrd(jn), domain_id=nidom )1306 & 1, jpi, 1, jpj, nit000, zjulian, rdt, nh_t(jn), nidtrd(jn), domain_id=nidom, snc4chunks=snc4set ) 1307 1307 1308 1308 !-- Define the ML depth variable … … 1317 1317 CALL dia_nam( clhstnam, nn_trd_trc, 'trdbio' ) 1318 1318 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 1319 & 1, jpi, 1, jpj, nit000, zjulian, rdt, nh_tb, nidtrdbio, domain_id=nidom )1319 & 1, jpi, 1, jpj, nit000, zjulian, rdt, nh_tb, nidtrdbio, domain_id=nidom, snc4chunks=snc4set ) 1320 1320 #endif 1321 1321 … … 1368 1368 !-- Leave IOIPSL/NetCDF define mode 1369 1369 DO jn = 1, jptra 1370 IF( ln_trdtrc(jn) ) CALL histend( nidtrd(jn) )1370 IF( ln_trdtrc(jn) ) CALL histend( nidtrd(jn), snc4set ) 1371 1371 END DO 1372 1372 1373 1373 #if defined key_lobster 1374 1374 !-- Leave IOIPSL/NetCDF define mode 1375 CALL histend( nidtrdbio )1375 CALL histend( nidtrdbio, snc4set ) 1376 1376 1377 1377 IF(lwp) WRITE(numout,*) -
branches/nemo_v3_3_beta/NEMOGCM/NEMO/TOP_SRC/trcdia.F90
r2287 r2364 176 176 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 177 177 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 178 & iiter, zjulian, zdt, nhorit5, nit5 , domain_id=nidom )178 & iiter, zjulian, zdt, nhorit5, nit5 , domain_id=nidom, snc4chunks=snc4set) 179 179 180 180 ! Vertical grid for tracer : gdept … … 197 197 198 198 ! end netcdf files header 199 CALL histend( nit5 )199 CALL histend( nit5, snc4set ) 200 200 IF(lwp) WRITE(numout,*) 201 201 IF(lwp) WRITE(numout,*) 'End of NetCDF Initialization in trcdit_wr' … … 303 303 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 304 304 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 305 & iiter, zjulian, zdt, nhoritd, nitd , domain_id=nidom )305 & iiter, zjulian, zdt, nhoritd, nitd , domain_id=nidom, snc4chunks=snc4set ) 306 306 307 307 ! Vertical grid for 2d and 3d arrays … … 332 332 333 333 ! CLOSE netcdf Files 334 CALL histend( nitd )334 CALL histend( nitd, snc4set ) 335 335 336 336 IF(lwp) WRITE(numout,*) … … 449 449 CALL histbeg( clhstnam, jpi, glamt, jpj, gphit, & 450 450 & iimi, iima-iimi+1, ijmi, ijma-ijmi+1, & 451 & iiter, zjulian, zdt, nhoritb, nitb , domain_id=nidom )451 & iiter, zjulian, zdt, nhoritb, nitb , domain_id=nidom, snc4chunks=snc4set ) 452 452 ! Vertical grid for biological trends 453 453 CALL histvert(nitb, 'deptht', 'Vertical T levels', 'm', ipk, gdept_0, ndepitb) … … 464 464 465 465 ! CLOSE netcdf Files 466 CALL histend( nitb )466 CALL histend( nitb, snc4set ) 467 467 468 468 IF(lwp) WRITE(numout,*)
Note: See TracChangeset
for help on using the changeset viewer.