Opened 23 months ago

Closed 23 months ago

Last modified 21 months ago

#2165 closed Enhancement (fixed)

Add deflation/chunking in REBUILD_NEMO

Reported by: mathiot Owned by: mathiot
Priority: low Milestone: 2018 release-4.0
Component: tools Version: trunk
Severity: minor Keywords: REBUILD_NEMO
Cc: Review:
MP ready?:



Rebuild eORCA025 with BGC (Medusa in this case) does not work (requirement in memory too large). To fix it, deflation and chunking has been added to rebuild_nemo (thanks Julien and Andrew). It appears also that latest netcdf version helps.

Tested on eORCA12 and eORCA025 + BGC on respectively 2394 and 1872 domains.

There is no change in the philosophy of rebuild_nemo itself, so the memory requirement is still large. All the details on compilation and usage are added in a README.rst.


New option for the rebuild_nemo script:

  NEMO Rebuild

  usage: rebuild_nemo [-l -p -s -m -n -r -d -x -y -z -t -c] filebase ndomain [rebuild dimensions]

  flags:    -l arch            submit to compute node
            -p num             use num threads
            -s num             split 4D vars into time slice of size num
            -m                 force masking of global arrays (zero if no mdi)
            -n namelist        full path to namelist file to be created (otherwise default nam_rebuild+_process_id is used)
            -r memory          Memory to request on compute node including units (Default = 10Gb)

      key_netcdf4 only 
            -d deflate_level     deflate level for output files
            -x chunksize along x 
            -y chunksize along y 
            -z chunksize along z 
            -t chunksize along t 
            -c total size of the chunk cache 

New namelist parameter for rebuild_nemo.exe:

l_maskout=.false       !(-m option: useful if input file comes from a run using land suppression)
nslicesize=0           !(-s option: 0 means no splitting in time slice)
deflate_level=1        !(-d option)
nc4_xchunk=206         !(-x option)
nc4_ychunk=135         !(-y option)
nc4_zchunk=1           !(-z option)
nc4_tchunk=1           !(-t option)
fchunksize=32000000    !(-c option)

To fully use the rebuild_name option (especially -l option), a param and rebuild_nemo_batch file has to be provided (below METO case as example) in the BATCH_TEMPLATES directory:

$: cat param_XC40_METO


$: cat rebuild_nemo_batch_XC40_METO

#!/bin/bash --login
#PBS -N Rebuild_Nemo
#PBS -q shared
#PBS -l ncpus=NTHREADS
#PBS -l mem=MEMORY
#PBS -l walltime=1:0:0


ulimit -s unlimited
ulimit -n NOPEN

module load cray-snplauncher
${indir}/rebuild_nemo.exe $nam_rebuild


exit $RC

Commit History (2)


Deleting branch used to add compression in rebuild_nemo (see ticket #2165). Added in tools at r10338.


Add deflation/chunking to rebuild_nemo tool. Fix #2165

Change History (2)

comment:1 Changed 23 months ago by mathiot

  • Resolution set to fixed
  • Status changed from new to closed

In 10338:

Add deflation/chunking to rebuild_nemo tool. Fix #2165

comment:2 Changed 21 months ago by mathiot

In 10488:

Deleting branch used to add compression in rebuild_nemo (see ticket #2165). Added in tools at r10338.

Note: See TracTickets for help on using tickets.