Opened 7 years ago

Closed 7 years ago

#336 closed defect (fixed)

On ada use nco/4.6.3 instead of nco/4.4.6 for ncrcat included in pack_output.

Reported by: mafoipsl Owned by: somebody
Priority: major Milestone: libIGCM_v2.8.4
Component: PostProcessing Version:
Keywords: pack_output nco ncrcat Cc:

Description

ncrcat ie nco/4.6.3 is producing too large files. We have to use nco/4.4.6

To do that we have to unload Intel2013_0.

Output files for 1Y IPSLCM6-MR :

  • with nco 4.4.6 : 303 GB
  • with nco 4.6.3 : 110 GB

Details for IPSLCM6-MR (LMDZ256x256x79eORCA1) files :

21G ATM/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_1D_histday.nc
49G ATM/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_1D_histday.nc

33G ATM/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_histmth.nc
95G ATM/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_histmth.nc

461M ATM/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_histmthNMC.nc
3.3G ATM/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_histmthNMC.nc

358M ICE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_icemod.nc
358M ICE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_icemod.nc

94K MBG/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_bioscalar.nc
91K MBG/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_bioscalar.nc

1.3G MBG/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_diad_T.nc
3.8G MBG/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_diad_T.nc

3.7G MBG/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_ptrc_T.nc
12G MBG/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_ptrc_T.nc

1.1M OCE/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_1D_scalar.nc
1.1M OCE/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_1D_scalar.nc

9.9G OCE/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_T.nc
30G OCE/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_T.nc

4.9G OCE/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_U.nc
15G OCE/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_U.nc

4.9G OCE/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_V.nc
15G OCE/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_V.nc

7.2G OCE/Output/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_W.nc
22G OCE/Output_nco_446/DA/CM6011.3-MR-pd-A04_19500101_19501231_5D_grid_W.nc

24M OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_diaptr.nc
24M OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_diaptr.nc

2.8G OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_T.nc
7.8G OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_T.nc

1.7G OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_U.nc
5.0G OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_U.nc

1.7G OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_V.nc
5.0G OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_V.nc

5.3G OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_W.nc
17G OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_grid_W.nc

244K OCE/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_scalar.nc
242K OCE/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_scalar.nc

3.6G SBG/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_stomate_history.nc
4.1G SBG/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_stomate_history.nc

146M SBG/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_stomate_ipcc_history.nc
146M SBG/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_stomate_ipcc_history.nc

417M SRF/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_sechiba_history_4dim.nc
835M SRF/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_sechiba_history_4dim.nc

1.6G SRF/Output/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_sechiba_history.nc
2.0G SRF/Output_nco_446/MO/CM6011.3-MR-pd-A04_19500101_19501231_1M_sechiba_history.nc

Change History (5)

comment:1 Changed 7 years ago by mafoipsl

With module load intel/2013.0 we have an error :

time ncrcat -C -p /workgpfs/rech/.../.../IGCM_OUT/IPSLCM6-MR/DEVT/pdControl/CM6011.3-MR-pd-A04/ATM/Output/MO CM6011.3-MR-pd-A04_19500101_19500131_1M_histmth.nc CM6011.3-MR-pd-A04_19500201_19500228_1M_histmth.nc CM6011.3-MR-pd-A04_19500301_19500331_1M_histmth.nc CM6011.3-MR-pd-A04_19500401_19500430_1M_histmth.nc CM6011.3-MR-pd-A04_19500501_19500531_1M_histmth.nc CM6011.3-MR-pd-A04_19500601_19500630_1M_histmth.nc CM6011.3-MR-pd-A04_19500701_19500731_1M_histmth.nc CM6011.3-MR-pd-A04_19500801_19500831_1M_histmth.nc CM6011.3-MR-pd-A04_19500901_19500930_1M_histmth.nc CM6011.3-MR-pd-A04_19501001_19501031_1M_histmth.nc CM6011.3-MR-pd-A04_19501101_19501130_1M_histmth.nc CM6011.3-MR-pd-A04_19501201_19501231_1M_histmth.nc --output CM6011.3-MR-pd-A04_19500101_19501231_1M_histmth_nco463.nc
/gpfs4l/smplocal/loadl/execute/ada337.idris.fr.1474243.0/MYJOBPACK3: line 48: 22061: Memory fault(coredump)

Tests done in WORKDIR rgzi003 ... /IPSLCM6-MR/IPSLCM6.0.11/T20170706/modipsl/config/IPSLCM6/CM6011.3-MR-pd-A04/POST_REDO

In libIGCM_sys_ada we need 3 different .atlas : execution in IBM mode, execution in Intel mode and execution of post-pro jobs.

Last edited 7 years ago by mafoipsl (previous) (diff)

comment:2 Changed 7 years ago by mafoipsl

One more thing:
on ada ncrcat with --buffer_size 838860800 takes more time. 7mn2 instead of 6mn44. It's acceptable.

comment:3 Changed 7 years ago by sdipsl

  • Milestone changed from libIGCM_v3 release candidate to libIGCM_v2.8.4

comment:4 Changed 7 years ago by sdipsl

done see r1409. run post-processing with intel environment

comment:5 Changed 7 years ago by sdipsl

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.