Changes between Version 6 and Version 7 of 2021WP/VLD-05_Coward_SETTE_inputs

Timestamp:

2021-03-15T18:46:42+01:00 (3 years ago)

Author:

acc

Comment:

--

2021WP/VLD-05_Coward_SETTE_inputs

@@ -515 +515 @@
 
 === Stage 2 details
+
+The second stage is to take the sets produced in stage 1 and to reduce the time period of any forcing data to more closely match that required for the standard SETTE testing. To permit some additional testing for those cases where longer tests may be required a period of twice the standard test length has been chosen. The standard test lengths are:
+
+||GYRE_PISCES ||90 days ||
+||ORCA2_ICE_PISCES ||62 days ||
+||ORCA2_OFF_PISCES ||95 days ||
+||AMM12 ||4 days ||
+||SAS ||16 days ||
+||ORCA2_ICE_OBS ||5 days ||
+||AGRIF ||5d and 9h ||
+||WED025 ||10 days ||
+
+where the maximum length returned by grepping the setting of ITEND from sette_reference_configurations.sh has been taken. There are two cases where simply doubling this length and selecting a number of time records based on that number does not work though:
+
+* ORCA2_OFF_PISCES which uses a climatological set of inputs and starts on 1st January => the last record is also needed at the start.
+* WED025 which starts on 15th January => 35 days of records are required for a 20 day test.
+
+More on these cases later but first the additional commands required to limit the time records are:
+
+{{{
+mkdir r4.2_RC
+cd r4.2_RC
+cp -pr ../r4.2_RC_FULL/AGRIF_DEMO_v4.2_RC_FULL   AGRIF_DEMO_v4.2_RC
+cp -pr ../r4.2_RC_FULL/AMM12_v4.2_RC_FULL        AMM12_v4.2_RC
+cp -pr ../r4.2_RC_FULL/ICE_AGRIF_v4.2_RC_FULL    ICE_AGRIF_v4.2_RC
+cp -pr ../r4.2_RC_FULL/ISOMIP+_v4.2_RC_FULL      ISOMIP+_v4.2_RC
+cp -pr ../r4.2_RC_FULL/ORCA2_ICE_v4.2_RC_FULL    ORCA2_ICE_v4.2_RC
+cp -pr ../r4.2_RC_FULL/ORCA2_OFF_v4.2_RC_FULL    ORCA2_OFF_v4.2_RC
+cp -pr ../r4.2_RC_FULL/SAS_v4.2_RC_FULL          SAS_v4.2_RC
+cp -pr ../r4.2_RC_FULL/WED025_v4.2_RC_FULL       WED025_v4.2_RC
+########## ORCA2_ICEcmds ##########
+cd ORCA2_ICE_v4.2_RC
+#
+#
+# ncar_precip.15JUNE2009_fill.nc ncar_rad.15JUNE2009_fill.nc slp.15JUNE2009_fill.nc
+#     LAT LON TIME
+# q_10.15JUNE2009_fill.nc t_10.15JUNE2009_fill.nc u_10.15JUNE2009_fill.nc v_10.15JUNE2009_fill.nc
+#     lat lon time
+# Need 180 days @ 6 hourly = 720 records
+#
+ncks --4 --dfl_lvl 1 --no_abc -v T_10_MOD -d time,0,719 --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1  t_10.15JUNE2009_fill.nc new_t_10.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v U_10_MOD -d time,0,719 --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1  u_10.15JUNE2009_fill.nc new_u_10.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v V_10_MOD -d time,0,719 --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1  v_10.15JUNE2009_fill.nc new_v_10.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v Q_10_MOD -d time,0,719 --cnk_plc='xpl' --cnk_dmn lon,32 --cnk_dmn lat,32 --cnk_dmn time,1  q_10.15JUNE2009_fill.nc new_q_10.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v LWDN_MOD,SWDN_MOD -d time,0,179 --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn time,1  ncar_rad.15JUNE2009_fill.nc new_ncar_rad.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v PRC_MOD1,SNOW --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn time,1  ncar_precip.15JUNE2009_fill.nc new_ncar_precip.15JUNE2009_fill.nc
+#
+ncks --4 --dfl_lvl 1 --no_abc -v SLP -d TIME,0,719 --cnk_plc='xpl' --cnk_dmn LON,32 --cnk_dmn LAT,32 --cnk_dmn TIME,1  slp.15JUNE2009_fill.nc new_slp.15JUNE2009_fill.nc
+#
+for f in new*; do ff=${f/new_}; mv $f $ff; done
+#
+ncks --4 --no_abc --cnk_plc='xpl' -d time_counter,0,719 --cnk_dmn x,30 --cnk_dmn y,30 --cnk_dmn time_counter,1  sdw_ecwaves_orca2.nc new_sdw_ecwaves_orca2.nc
+mv new_sdw_ecwaves_orca2.nc sdw_ecwaves_orca2.nc
+cd ../
+# WEDcmds
+cd WED025_v4.2_RC
+# u10 v10 t10 precip rsds q10 slp rlds snow
+# WED025 starts in middle of January; need 20 days from then = 35 days = 280 records @ 3hourly
+for v in u10 v10 t10 precip rsds q10 slp rlds snow
+do
+ ncks -O -d time,0,279 --cnk_dmn longitude,64 --cnk_dmn latitude,64 ${v}_JRA_y2000.nc 20d_${v}_JRA_y2000.nc
+done
+#
+for v in u10 v10 t10 precip rsds q10 slp rlds snow
+do
+ mv 20d_${v}_JRA_y2000.nc ${v}_JRA_y2000.nc
+done
+}}}
+These changes reduce the overall volume from 5.5GB to 4.5GB but the two cases mentioned earlier are still large contributors to this volume. The next stage will reduce the volume further by reducing the number of significant digits used for most data. This will allow for much more efficient compression but the SETTE results will be changed as a result. There is, however, an intermediate option which is to reduce the precision of the data that has to be supplied in order to maintain contiguous time records but which are not actually used. 
+
+* In the ORCA2_OFF_PISCES case records 0-37 and 72 of each 5-day mean, annual dataset need to be kept at the original precision but all other records can be reduced to the minimum precision.
+
+* For WED025, records 0-110 are not used and can, therefore be held at minimum precision.
+
+Using ncks to achieve is troublesome with compressed data. I appears to be reluctant to insert later records into
+compressed datasets. However, carrying out all operations on netCDF-3 format files and converting to netCDF-4 at the end works. Thus, this rather tortuous combination achieves the desired result:
+
+{{{
+#!/bin/bash
+if [ 1 == 1 ] ; then
+cd ORCA2_OFF_v4.2_RC
+for var in T
+do
+ echo ${var}
+ # Create minimal precision netcdf3 version of entire dataset
+   ncks -3 -O --no_abc --ppc default=1 dyna_grid_${var}.nc new2_dyna_grid_${var}.nc
+ # Create netcdf3 version of original dataset
+   ncks -3 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ # Overwrite records 0-71 with minimal precision
+   ncks -3 --no_abc -A -d time_counter,0,71 new2_dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ # Overwrite records 0-37 with original precision
+   ncks -3 --no_abc -A -d time_counter,0,37 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ # Create netcdf-4 final version with required chunking and compression
+   nccopy -7 -c "time_counter/1,x/60,y/50,deptht/4" -d 4 new3_dyna_grid_${var}.nc new4_dyna_grid_${var}.nc
+done
+# Tidy up
+rm new[2,3]*.nc
+mv new4_dyna_grid_T.nc dyna_grid_T.nc
+
+for var in U
+do
+ echo ${var}
+ ncks -3 -O --no_abc --ppc default=1 dyna_grid_${var}.nc new2_dyna_grid_${var}.nc
+ ncks -3 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,71 new2_dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,37 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ nccopy -7 -c "time_counter/1,x/60,y/50,depthu/4" -d 4 new3_dyna_grid_${var}.nc new4_dyna_grid_${var}.nc
+done
+rm new[2,3]*.nc
+mv new4_dyna_grid_U.nc dyna_grid_U.nc
+
+for var in V
+do
+ echo ${var}
+ ncks -3 -O --no_abc --ppc default=1 dyna_grid_${var}.nc new2_dyna_grid_${var}.nc
+ ncks -3 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,71 new2_dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,37 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ nccopy -7 -c "time_counter/1,x/60,y/50,depthv/4" -d 4 new3_dyna_grid_${var}.nc new4_dyna_grid_${var}.nc
+done
+rm new[2,3]*.nc
+mv new4_dyna_grid_V.nc dyna_grid_V.nc
+
+for var in W
+do
+ echo ${var}
+ ncks -3 -O --no_abc --ppc default=1 dyna_grid_${var}.nc new2_dyna_grid_${var}.nc
+ ncks -3 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,71 new2_dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ ncks -3 --no_abc -A -d time_counter,0,37 dyna_grid_${var}.nc new3_dyna_grid_${var}.nc
+ nccopy -7 -c "time_counter/1,x/60,y/50,depthw/4" -d 4 new3_dyna_grid_${var}.nc new4_dyna_grid_${var}.nc
+done
+rm new[2,3]*.nc
+mv new4_dyna_grid_W.nc dyna_grid_W.nc
+cd ../
+fi
+#
+if [ 1 == 1 ] ; then
+cd WED025_v4.2_RC
+# WEDcmds
+# u10 v10 t10 precip rsds q10 slp rlds snow
+# WED025 starts in middle of January; need 20 days from then = 35 days = 280 records @ 3hourly
+# But the first 13.875 days records (111 @ 3 hourly) can be stored at minimal precision
+for v in u10 v10 t10 precip rsds q10 slp rlds snow
+do
+ ncks -3 -O --no_abc --ppc default=1 ${v}_JRA_y2000.nc new2_${v}_JRA_y2000.nc
+ ncks -3 ${v}_JRA_y2000.nc new3_${v}_JRA_y2000.nc
+ ncks -3 --no_abc -A -d time,0,110 new2_${v}_JRA_y2000.nc new3_${v}_JRA_y2000.nc
+ nccopy -7 -c "time/1,longitude/64,latitude/64" -d 4 new3_${v}_JRA_y2000.nc new4_${v}_JRA_y2000.nc
+done
+rm new[2,3]*.nc
+#
+for v in u10 v10 t10 precip rsds q10 slp rlds snow
+do
+ mv new4_${v}_JRA_y2000.nc ${v}_JRA_y2000.nc
+done
+#
+cd ../
+fi
+}}}
+
+With this secondary manipulation the overall volumes reduces from 4.5 GB to 3.7GB and there is no change in SETTE results when compared with results using the r4.2_RC_FULL set
+
+
 === Documentation updates