| Version 6 (modified by acc, 3 years ago) (diff) |
|---|
VLD-05_Coward_SETTE_inputs
Last edition: Wikinfo(changed_ts)? by Wikinfo(changed_by)?
The PI is responsible to closely follow the progress of the action, and especially to contact NEMO project manager if the delay on preview (or review) are longer than the 2 weeks expected.
Summary
| Action | VLD-05_Coward_SETTE_inputs |
|---|---|
| PI(S) | Andrew Coward |
| Digest | Rationalisation of SETTE inputs |
| Dependencies | If any |
| Branch | N/A |
| Previewer(s) | Names |
| Reviewer(s) | Names |
| Ticket | #2637 |
Description
Collection and rationalisation of SETTE inputs. The set of input files for the full suite of SETTE tests has evolved rapidly to keep pace with changes to the code (such as the removal of haloes from external files). The current set needs to be cleaned of unused data and chunked and compressed with setting consistent with exascale ambitions. A definitive set then needs to be hosted in a publically available location with a fixed DOI. A lighter version (for example, reduced time-levels in forcing data) may also be appropriate for future containerisation or cloud-deployment of testing services.
Stage 1: Collect all input file sets for SETTE 4.2_RC (current trunk). Confirm successful SETTE results and store reference set of SETTE results. Remove redundant files and variables from contents. Make sure sensible chunking and compression choices have been made in all cases. Produce clean 'r4.2_RC_FULL' set. See details here
Stage 2: Reduce data volumes by selecting only sufficient forcing data for twice the period of each standard test. Run SETTE with reduced set and confirm unchanged results. Create recommended 'r4.2_RC' set. See details here
Stage3: Reduce data volumes further by limiting number of significant digits in all fields other than domain and coordinate variables. Confirm SETTE tests are still successful (results WILL be different). Create optional 'r4.2_RC_LITE' set.
Stage 4: Find a hosting and distribution option
Stage 5: Document and archive scripts so that the process can be repeated if(when) changes are made to the FULL set.
...
Implementation
Stage 1 details
Stage 1 involves gathering all SETTE input files and systematically checking contents for redundancy and opportunities for compression. Where chunking and compression have already been applied, it is also important to check the current settings for validity. Here is a typical example from the AGRIF_DEMO_v4.x.tar set:
nccnkrpt ORCA_R2_zps_domcfg_agrif.nc
t = UNLIMITED ; // (1 currently)
x = 180 ;
y = 148 ;
z = 31 ;
double time_counter(t) _ChunkSizes = 512 ;
double glamt(t, y, x) _ChunkSizes = 4, 148, 180 ;
double glamu(t, y, x) _ChunkSizes = 4, 148, 180 ;
double glamv(t, y, x) _ChunkSizes = 4, 148, 180 ;
double glamf(t, y, x) _ChunkSizes = 4, 148, 180 ;
double gphit(t, y, x) _ChunkSizes = 4, 148, 180 ;
double gphiu(t, y, x) _ChunkSizes = 4, 148, 180 ;
double gphiv(t, y, x) _ChunkSizes = 4, 148, 180 ;
double gphif(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e1t(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e1u(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e1v(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e1f(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e2t(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e2u(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e2v(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e2f(t, y, x) _ChunkSizes = 4, 148, 180 ;
double ff_f(t, y, x) _ChunkSizes = 4, 148, 180 ;
double ff_t(t, y, x) _ChunkSizes = 4, 148, 180 ;
double e3t_1d(t, z) _ChunkSizes = 1, 31 ;
double e3w_1d(t, z) _ChunkSizes = 1, 31 ;
double e3t_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3u_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3v_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3f_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3w_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3uw_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
double e3vw_0(t, z, y, x) _ChunkSizes = 1, 31, 148, 180 ;
int bottom_level(t, y, x) _ChunkSizes = 6, 148, 180 ;
int top_level(t, y, x) _ChunkSizes = 6, 148, 180 ;
float bathy_metry(t, y, x) _ChunkSizes = 6, 148, 180 ;
where nccnkrpt is my bash function defined as:
function nccnkrpt { if [ `ncdump -k ${1} | awk '{print $1}'` == "netCDF-4" ] ; then ncks --cdl -m ${1} | grep '=' | grep -v ":" ; ncdump -s -h ${1} | grep -e ") ;" -e _ChunkSizes | sed -e 's/.*:/\t/' | sed -e ':x /) ;$/ { N; s/;\n//g ; bx }'; fi ; }
which helps to reduce the verbosity of the ncdump -s -h output into a more digestible form. In this example the dataset has already been chunked and compressed but the chunksizes are an odd choice. Having chunk sizes which span the entire dataset will restrict future scalability since any access to these data will require reading and uncompressing these large chunks irrespective of the size of the calling domain. A chunksize greater than 1 for the t dimension is also wasteful and confusing given that the file only contains a single time-level.
The 'right' choice for chunksizes is somewhat arbitrary but given our exascale ambitions of efficient performance with processor domains of O(10x10) in size, a target chunk-size around 64x64 would seem a reasonable compromise. Chunk-sizes which are too small will compromise compressibility and require more chunk meta-data in the file. Sizes which are too large will affect scalability and cause unnecessary delays at start-up. For the ORCA2 domain a chunk-size of 60x50 is chosen since this also avoids any underpopulated chunks. For the vertical dimension, I have used a chunk-size of 4 but there is probably little gain here since the volume data is always read as a whole and a full-depth chunk is equally appropriate. Breaking the vertical dimension into smaller chunks, however, may help other applications that use the domain configuration file and only wish to select specific levels.
All the other files in the original AGRIF_DEMO set are in classic NetCDF3 format and therefore unchunked and uncompressed. All these files have been converted to NetCDF4 with suitable chunk-size choices. The actual sizes vary slightly for each of the 3 AGRIF-level sets so as to avoid under-populated chunks.
| ORCA_R2_zps_domcfg_agrif.nc | 1_ORCA_R2_zps_domcfg_agrif.nc | 2_ORCA_R05_zps_domcfg_agrif.nc | 3_ORCA_R017_zps_domcfg_agrif.nc |
| t = UNLIMITED ( 1 ) | t = UNLIMITED ( 1 ) | t = UNLIMITED ( 1 ) | t = UNLIMITED ( 1 ) |
| x = 180 | x = 48 | x = 132 | x = 134 |
| y = 148 | y = 50 | y = 140 | y = 128 |
| z = 31 | z = 31 | z = 31 | z = 31 |
The complete set of ncks commands used to create the r4.2_RC_FULL set for AGRIF_DEMO_r4.2_RC_FULL from a copy of AGRIF_DEMO_v4.x is:
# AGRIFcmds
#### Mother grid ####
#
ncks --no_abc --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn z,4 ORCA_R2_zps_domcfg_agrif.nc new_ORCA_R2_zps_domcfg_agrif.nc
mv new_ORCA_R2_zps_domcfg_agrif.nc ORCA_R2_zps_domcfg_agrif.nc
#
#### Nest level 1 ####
#
for f in 1_chlorophyll.nc 1_geothermal_heating.nc 1_runoff_core_monthly.nc 1_sss_data.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,48 --cnk_dmn y,50 $f new_$f
done
#
for f in 1_data_1m_potential_temperature_nomask.nc 1_data_1m_salinity_nomask.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,48 --cnk_dmn y,50 --cnk_dmn z,4 $f new_$f
done
#
for f in 1_eddy_viscosity_3D.nc 1_ORCA_R2_zps_domcfg_agrif.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,48 --cnk_dmn y,50 --cnk_dmn z,4 $f new_$f
done
#
for f in 1_resto.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn x,48 --cnk_dmn y,50 --cnk_dmn z,4 $f new_$f
done
#
for f in 1_weights_core_orca2_bicubic_noc.nc 1_weights_core_orca2_bilinear_noc.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn lon,48 --cnk_dmn lat,50 $f new_$f
done
#
#### Nest level 2 ####
#
for f in 2_chlorophyll.nc 2_geothermal_heating.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,66 --cnk_dmn y,70 $f new_$f
done
#
for f in 2_data_1m_potential_temperature_nomask.nc 2_data_1m_salinity_nomask.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,66 --cnk_dmn y,70 --cnk_dmn z,4 $f new_$f
done
#
for f in 2_ORCA_R05_zps_domcfg_agrif.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,66 --cnk_dmn y,70 --cnk_dmn z,4 $f new_$f
done
#
for f in 2_weights_core2_nordic1_bicub.nc 2_weights_core2_nordic1_bilin.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn lon,66 --cnk_dmn lat,70 $f new_$f
done
#
#### Nest level 3 ####
#
for f in 3_chlorophyll.nc 3_geothermal_heating.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,67 --cnk_dmn y,64 $f new_$f
done
#
for f in 3_data_1m_potential_temperature_nomask.nc 3_data_1m_salinity_nomask.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,67 --cnk_dmn y,64 --cnk_dmn z,4 $f new_$f
done
#
for f in 3_ORCA_R017_zps_domcfg_agrif.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,67 --cnk_dmn y,64 --cnk_dmn z,4 $f new_$f
done
#
for f in 3_weights_core2_nordic2_bicub.nc 3_weights_core2_nordic2_bilin.nc
do
ncks --no_abc --4 --dfl_lvl 3 --cnk_plc='xpl' --cnk_dmn lon,67 --cnk_dmn lat,64 $f new_$f
done
#
for f in new*; do ff=${f/new_}; mv $f $ff; done
#
The process is similar for the other configurations based on the following original sets:
mkdir -p ../r4.2_RC_FULL cp -pr AGRIF_DEMO_v4.x ../r4.2_RC_FULL/AGRIF_DEMO_v4.2_RC_FULL cp -pr AMM12_v4.0 ../r4.2_RC_FULL/AMM12_v4.2_RC_FULL cp -pr ICE_AGRIF_v4.x ../r4.2_RC_FULL/ICE_AGRIF_v4.2_RC_FULL cp -pr ISOMIP+_v4.0 ../r4.2_RC_FULL/ISOMIP+_v4.2_RC_FULL cp -pr ORCA2_ICE_v4.x ../r4.2_RC_FULL/ORCA2_ICE_v4.2_RC_FULL cp -pr ORCA2_OFF_v4.x ../r4.2_RC_FULL/ORCA2_OFF_v4.2_RC_FULL cp -pr SAS_v4.x ../r4.2_RC_FULL/SAS_v4.2_RC_FULL cp -pr WED025_v4.2 ../r4.2_RC_FULL/WED025_v4.2_RC_FULL
It is worth noting a few redundant files that can simply be removed:
# # Remove missing (and erroneously placed) links # rm AMM12_r4.2_RC_FULL/bdydta/bdydta rm AMM12_r4.2_RC_FULL/fluxes/fluxes # # Remove old, unused versions # rm ORCA2_ICE_r4.2_RC_FULL/weights_core2_orca2_bicub.nc.old rm ORCA2_ICE_r4.2_RC_FULL/weights_core2_orca2_bilin.nc.old # # Remove old, unused versions superceded by domain configuration files # rm WED025_r4.2_RC_FULL/bathy_meter_WED025.nc rm WED025_r4.2_RC_FULL/coordinates_WED025.nc
And a few selected oddities in the original chunk settings that have been corrected in this tidy up:
nccnkrpt ICE_AGRIF_v4.x/initice.nc
time_counter = UNLIMITED ; // (1 currently)
x = 97 ;
y = 97 ; |
float nav_lon(y, x) _ChunkSizes = 97, 97 ; |
float nav_lat(y, x) _ChunkSizes = 97, 97 ; V
float time_counter(time_counter) _ChunkSizes = 1048576 ;
float ati(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
float hti(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
float hts(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
float smi(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
float tmi(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
float tsu(time_counter, y, x) _ChunkSizes = 1, 97, 97 ;
nccnkrpt ORCA2_ICE_v4.x/sss_data.nc
time_counter = UNLIMITED ; // (12 currently)
x = 180 ;
y = 148 ;
float nav_lat(y, x) _ChunkSizes = 148, 180 ; |
float nav_lon(y, x) _ChunkSizes = 148, 180 ; V
float time_counter(time_counter) _ChunkSizes = 1024 ;
float sss(time_counter, y, x) _ChunkSizes = 21, 148, 180 ;
^
|
nccnkrpt ORCA2_OFF_v4.x/dyna_grid_T.nc
deptht = 31 ;
time_counter = UNLIMITED ; // (73 currently)
x = 180 ;
y = 148 ;
float nav_lon(y, x) _ChunkSizes = 148, 180 ;
float nav_lat(y, x) _ChunkSizes = 148, 180 ;
float deptht(deptht) _ChunkSizes = 31 ;
float time_counter(time_counter) _ChunkSizes = 1024 ;
float votemper(time_counter, deptht, y, x) _ChunkSizes = 1, 31, 148, 180 ;
float vosaline(time_counter, deptht, y, x) _ChunkSizes = 1, 31, 148, 180 ;
float sosstsst(time_counter, y, x) _ChunkSizes = 54, 126, 154 ;
float sosaline(time_counter, y, x) _ChunkSizes = 54, 126, 154 ;
float sossheig(time_counter, y, x) _ChunkSizes = 54, 126, 154 ;
float iowaflup(time_counter, y, x) _ChunkSizes = 54, 126, 154
??
The full set of commands used to create the r4.2_RC_FULL set is (in additional to those already shown):
########## AMMcmds ########## cd AMM12_v4.2_RC_FULL ncks --no_abc --cnk_plc='xpl' --cnk_dmn x,32 --cnk_dmn y,32 --cnk_dmn z,6 --cnk_dmn t,1 --4 --dfl_lvl 3 amm12_restart_oce.nc new_amm12_restart_oce.nc mv new_amm12_restart_oce.nc amm12_restart_oce.nc # ncks --4 --no_abc --cnk_dmn=x,32 --cnk_dmn=y,32 --cnk_dmn=z,6 --cnk_dmn=t,1 --dfl_lvl 3 --cnk_plc='xpl' AMM_R12_sco_domcfg.nc new_AMM_R12_sco_domcfg.nc mv new_AMM_R12_sco_domcfg.nc AMM_R12_sco_domcfg.nc # ncks --no_abc --cnk_plc='xpl' --cnk_dmn x,32 --cnk_dmn y,32 --cnk_dmn time_counter,1 --4 --dfl_lvl 3 amm12_rivers.nc new_amm12_rivers.nc mv new_amm12_rivers.nc amm12_rivers.nc # cd fluxes/ mkdir new for f in *.nc; do ncks --no_abc --cnk_plc='xpl' --cnk_dmn x,32 --cnk_dmn y,32 --cnk_dmn t,1 --4 --dfl_lvl 3 $f new/$f; done cd new mv amm* ../ cd ../ rmdir new cd ../ # cd bdydta mkdir new for f in amm12_bdyT_tra*; do ncks --no_abc --cnk_plc='xpl' --cnk_dmn x,64 --cnk_dmn y,1 --cnk_dmn=deptht,51 --cnk_dmn time_counter,1 --4 --dfl_lvl 3 $f new/$f; done cd new mv amm* ../ cd ../ rmdir new cd ../ # cd ../
########## ICEcmds ########## cd ICE_AGRIF_v4.2_RC_FULL ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,49 --cnk_dmn y,49 initice.nc new_initice.nc mv new_initice.nc initice.nc cd ../
########## ISOcmds ########## cd ISOMIP+_v4.2_RC_FULL ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,42 --cnk_dmn nav_lev,1 --cnk_dmn time_counter,1 domain_cfg.nc new_domain_cfg.nc ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,21 --cnk_dmn t,1 isomip+_NEMO_242_geom_ocean3.nc new_isomip+_NEMO_242_geom_ocean3.nc ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,21 --cnk_dmn t,1 isomip+_NEMO_242_geom_ocean4.nc new_isomip+_NEMO_242_geom_ocean4.nc ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,21 --cnk_dmn z,10 nemo_base_COLD.nc new_nemo_base_COLD.nc ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,21 --cnk_dmn z,10 nemo_base_WARM.nc new_nemo_base_WARM.nc ncks --no_abc --cnk_dmn x,52 --cnk_dmn y,21 --cnk_dmn z,10 resto.nc new_resto.nc mv new_domain_cfg.nc domain_cfg.nc mv new_isomip+_NEMO_242_geom_ocean3.nc isomip+_NEMO_242_geom_ocean3.nc mv new_isomip+_NEMO_242_geom_ocean4.nc isomip+_NEMO_242_geom_ocean4.nc mv new_nemo_base_COLD.nc nemo_base_COLD.nc mv new_nemo_base_WARM.nc nemo_base_WARM.nc mv new_resto.nc resto.nc cd ../
########## ORCA2_ICEcmds ##########
cd ORCA2_ICE_v4.2_RC_FULL
ncks --no_abc --cnk_plc='xpl' --cnk_dmn lon,30 --cnk_dmn lat,30 weights_core2_orca2_bicub.nc new_weights_core2_orca2_bicub.nc
ncks --no_abc --cnk_plc='xpl' --cnk_dmn lon,30 --cnk_dmn lat,30 weights_core2_orca2_bilin.nc new_weights_core2_orca2_bilin.nc
mv new_weights_core2_orca2_bilin.nc weights_core2_orca2_bilin.nc
mv new_weights_core2_orca2_bicub.nc weights_core2_orca2_bicub.nc
#
# ncar_precip.15JUNE2009_fill.nc ncar_rad.15JUNE2009_fill.nc slp.15JUNE2009_fill.nc
# LAT LON TIME
# q_10.15JUNE2009_fill.nc t_10.15JUNE2009_fill.nc u_10.15JUNE2009_fill.nc v_10.15JUNE2009_fill.nc
# lat lon time
#
# Note CORE forcing datasets contain some redundant variables; only keep those actually used
#
ncks --4 --dfl_lvl 1 --no_abc -v T_10_MOD --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1 t_10.15JUNE2009_fill.nc new_t_10.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v U_10_MOD --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1 u_10.15JUNE2009_fill.nc new_u_10.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v V_10_MOD --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1 v_10.15JUNE2009_fill.nc new_v_10.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v Q_10_MOD --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1 q_10.15JUNE2009_fill.nc new_q_10.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v LWDN_MOD,SWDN_MOD --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn time,1 ncar_rad.15JUNE2009_fill.nc new_ncar_rad.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v PRC_MOD1,SNOW --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn time,1 ncar_precip.15JUNE2009_fill.nc new_ncar_precip.15JUNE2009_fill.nc
#
ncks --4 --dfl_lvl 1 --no_abc -v SLP --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn TIME,1 slp.15JUNE2009_fill.nc new_slp.15JUNE2009_fill.nc
#
for f in new*; do ff=${f/new_}; mv $f $ff; done
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 sdw_ecwaves_orca2.nc new_sdw_ecwaves_orca2.nc
mv new_sdw_ecwaves_orca2.nc sdw_ecwaves_orca2.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn z,4 --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 data_1m_potential_temperature_nomask.nc new_data_1m_potential_temperature_nomask.nc
mv new_data_1m_potential_temperature_nomask.nc data_1m_potential_temperature_nomask.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn z,4 --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 data_1m_salinity_nomask.nc new_data_1m_salinity_nomask.nc
mv new_data_1m_salinity_nomask.nc data_1m_salinity_nomask.nc
#
ncks --no_abc --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn z,4 ORCA_R2_zps_domcfg.nc new_ORCA_R2_zps_domcfg.nc
mv new_ORCA_R2_zps_domcfg.nc ORCA_R2_zps_domcfg.nc
#
ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 int_wave_mix.nc new_int_wave_mix.nc
mv new_int_wave_mix.nc int_wave_mix.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn z,4 --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 sali_ref_clim_monthly.nc new_sali_ref_clim_monthly.nc
mv new_sali_ref_clim_monthly.nc sali_ref_clim_monthly.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 runoff_core_monthly.nc new_runoff_core_monthly.nc
mv new_runoff_core_monthly.nc runoff_core_monthly.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 sst_data.nc new_sst_data.nc
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 sss_data.nc new_sss_data.nc
mv new_sst_data.nc sst_data.nc
mv new_sss_data.nc sss_data.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn X1,30 --cnk_dmn Y1,30 --cnk_dmn time_counter,1 subbasins.nc new_subbasins.nc
mv new_subbasins.nc subbasins.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 slaReferenceLevel.nc new_slaReferenceLevel.nc
mv new_slaReferenceLevel.nc slaReferenceLevel.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn z,4 resto.nc new_resto.nc
mv new_resto.nc resto.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn d01,30 --cnk_dmn d02,30 mask_itf.nc new_mask_itf.nc
mv new_mask_itf.nc mask_itf.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn z,4 --cnk_dmn t,1 eddy_viscosity_2D.nc new_eddy_viscosity_2D.nc
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn z,4 --cnk_dmn t,1 eddy_viscosity_3D.nc new_eddy_viscosity_3D.nc
mv new_eddy_viscosity_2D.nc eddy_viscosity_2D.nc
mv new_eddy_viscosity_3D.nc eddy_viscosity_3D.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn t,1 geothermal_heating.nc new_geothermal_heating.nc
mv new_geothermal_heating.nc geothermal_heating.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 K1rowdrg.nc new_K1rowdrg.nc
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 M2rowdrg.nc new_M2rowdrg.nc
mv new_K1rowdrg.nc K1rowdrg.nc
mv new_M2rowdrg.nc M2rowdrg.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 calving.nc new_calving.nc
mv new_calving.nc calving.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 chlorophyll.nc new_chlorophyll.nc
mv new_chlorophyll.nc chlorophyll.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1 --cnk_dmn z,4 assim_background_increments.nc new_assim_background_increments.nc
mv new_assim_background_increments.nc assim_background_increments.nc
#
ncks --4 --no_abc --cnk_plc='xpl' --cnk_dmn x,30 --cnk_dmn y,30 ahmcoef.nc new_ahmcoef.nc
mv new_ahmcoef.nc ahmcoef.nc
#
cd ../
########## ORCA2_OFFcmds ########## cd ORCA2_OFF_v4.2_RC_FULL ncks --no_abc --cnk_plc='xpl' --cnk_dmn t,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn z,4 ORCA_R2_zps_domcfg.nc new_ORCA_R2_zps_domcfg.nc mv new_ORCA_R2_zps_domcfg.nc ORCA_R2_zps_domcfg.nc # ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn deptht,4 dyna_grid_T.nc new_dyna_grid_T.nc # ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn depthu,4 dyna_grid_U.nc new_dyna_grid_U.nc # ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn depthv,4 dyna_grid_V.nc new_dyna_grid_V.nc # ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn depthw,4 dyna_grid_W.nc new_dyna_grid_W.nc # mv new_dyna_grid_T.nc dyna_grid_T.nc mv new_dyna_grid_U.nc dyna_grid_U.nc mv new_dyna_grid_V.nc dyna_grid_V.nc mv new_dyna_grid_W.nc dyna_grid_W.nc # cd ../
########## SAScmds ########## cd SAS_v4.2_RC_FULL ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn deptht,4 sas_grid_T.nc new_sas_grid_T.nc ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn depthu,4 sas_grid_U.nc new_sas_grid_U.nc ncks --no_abc --cnk_plc='xpl' --cnk_dmn time_counter,1 --cnk_dmn x,60 --cnk_dmn y,50 --cnk_dmn depthv,4 sas_grid_V.nc new_sas_grid_V.nc mv new_sas_grid_T.nc sas_grid_T.nc mv new_sas_grid_U.nc sas_grid_U.nc mv new_sas_grid_V.nc sas_grid_V.nc cd ../
########## WEDcmds ##########
cd WED025_v4.2_RC_FULL
# u10 v10 t10 precip rsds q10 slp rlds snow
# WED025 starts in middle of January; need 20 days from then = 35 days = 280 records @ 3hourly
for v in u10 v10 t10 precip rsds q10 slp rlds snow
do
ncks --cnk_dmn longitude,64 --cnk_dmn latitude,64 ${v}_JRA_y2000.nc 20d_${v}_JRA_y2000.nc
done
#
for v in u10 v10 t10 precip rsds q10 slp rlds snow
do
mv 20d_${v}_JRA_y2000.nc ${v}_JRA_y2000.nc
done
#
ncks --cnk_dmn xbT,50 WED025_bdyT_tra_y1999.nc new_WED025_bdyT_tra_y1999.nc
ncks --cnk_dmn xbT,50 WED025_bdyT_tra_y2000.nc new_WED025_bdyT_tra_y2000.nc
ncks --cnk_dmn xbU,50 WED025_bdyU_u3d_y1999.nc new_WED025_bdyU_u3d_y1999.nc
ncks --cnk_dmn xbU,50 WED025_bdyU_u3d_y2000.nc new_WED025_bdyU_u3d_y2000.nc
ncks --cnk_dmn xbV,44 WED025_bdyV_u3d_y1999.nc new_WED025_bdyV_u3d_y1999.nc
ncks --cnk_dmn xbV,44 WED025_bdyV_u3d_y2000.nc new_WED025_bdyV_u3d_y2000.nc
mv new_WED025_bdyT_tra_y1999.nc WED025_bdyT_tra_y1999.nc
mv new_WED025_bdyT_tra_y2000.nc WED025_bdyT_tra_y2000.nc
mv new_WED025_bdyU_u3d_y1999.nc WED025_bdyU_u3d_y1999.nc
mv new_WED025_bdyU_u3d_y2000.nc WED025_bdyU_u3d_y2000.nc
mv new_WED025_bdyV_u3d_y1999.nc WED025_bdyV_u3d_y1999.nc
mv new_WED025_bdyV_u3d_y2000.nc WED025_bdyV_u3d_y2000.nc
ncks --cnk_dmn time_counter,1 WED025_bdyT_tra_y1999.nc new_WED025_bdyT_tra_y1999.nc
ncks --cnk_dmn time_counter,1 WED025_bdyT_tra_y2000.nc new_WED025_bdyT_tra_y2000.nc
ncks --cnk_dmn time_counter,1 WED025_bdyU_u3d_y1999.nc new_WED025_bdyU_u3d_y1999.nc
ncks --cnk_dmn time_counter,1 WED025_bdyU_u3d_y2000.nc new_WED025_bdyU_u3d_y2000.nc
ncks --cnk_dmn time_counter,1 WED025_bdyV_u3d_y1999.nc new_WED025_bdyV_u3d_y1999.nc
ncks --cnk_dmn time_counter,1 WED025_bdyV_u3d_y2000.nc new_WED025_bdyV_u3d_y2000.nc
mv new_WED025_bdyT_tra_y1999.nc WED025_bdyT_tra_y1999.nc
mv new_WED025_bdyT_tra_y2000.nc WED025_bdyT_tra_y2000.nc
mv new_WED025_bdyU_u3d_y1999.nc WED025_bdyU_u3d_y1999.nc
mv new_WED025_bdyU_u3d_y2000.nc WED025_bdyU_u3d_y2000.nc
mv new_WED025_bdyV_u3d_y1999.nc WED025_bdyV_u3d_y1999.nc
mv new_WED025_bdyV_u3d_y2000.nc WED025_bdyV_u3d_y2000.nc
#
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 domain_cfg.nc new_domain_cfg.nc
mv new_domain_cfg.nc domain_cfg.nc
#
ncks --no_abc --cnk_dmn lon,64 --cnk_dmn lat,64 weights_bilin_JRA.nc new_weights_bilin_JRA.nc
ncks --no_abc --cnk_dmn lon,64 --cnk_dmn lat,64 weights_bicubic_JRA.nc new_weights_bicubic_JRA.nc
mv new_weights_bilin_JRA.nc weights_bilin_JRA.nc
mv new_weights_bicubic_JRA.nc weights_bicubic_JRA.nc
#
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 --cnk_dmn time_counter,1 WED025_icb_y1999.nc new_WED025_icb_y1999.nc
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 --cnk_dmn time_counter,1 WED025_icb_y2000.nc new_WED025_icb_y2000.nc
mv new_WED025_icb_y1999.nc WED025_icb_y1999.nc
mv new_WED025_icb_y2000.nc WED025_icb_y2000.nc
#
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 --cnk_dmn time_counter,1 --cnk_dmn z,15 WED025_init_JRA_200001.nc new_WED025_init_JRA_200001.nc
mv new_WED025_init_JRA_200001.nc WED025_init_JRA_200001.nc
#
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 --cnk_dmn time_counter,1 chlorophyll_WED025.nc new_chlorophyll_WED025.nc
mv new_chlorophyll_WED025.nc chlorophyll_WED025.nc
#
ncks --no_abc --cnk_dmn x,64 --cnk_dmn y,64 --cnk_dmn time_counter,1 isfmlt_par.nc new_isfmlt_par.nc
mv new_isfmlt_par.nc isfmlt_par.nc
#
ncks --no_abc --cnk_dmn xbt,50 --cnk_dmn xbu,50 --cnk_dmn xbv,44 coordinates_bdy_WED025.nc new_coordinates_bdy_WED025.nc
mv new_coordinates_bdy_WED025.nc coordinates_bdy_WED025.nc
#
cd ../
#
The files created by all these commands have been fully SETTE tested with trunk revision 14595 and produce identical results to the same tests performed with the original files.
Stage 2 details
Documentation updates
...
Preview
...
Tests
...
Review
...
