LMDZOR_v4 configuration

Josefine Ghattas

LMDZOR_v4 is the current version using LMDZ and ORCHIDEE. It can be used with LMDZ4 production version LMDZ4_AR5 or with one of the latest versions of LMDZ5/trunk. Therefore before extracting the model think about which version of LMDZ you want and correct eventually in modipsl/util/mod.def. Before running this model set up, choose between 2 mode : Interannual AMIP simulation or climatological simulation. No default setup exists. Modifications done by the user in the lmdz.card or orchidee.card are often necessary to adapt to a specific simulation. Modifications in lmdz.driver or orchidee.driver are less often necessary.

In this page you will find information specific to the configuration LMDZOR_v4. This page does not intend to explain how to use LMDZOR_v4 configuration! To learn more about using the configurations in modipsl see => platform/documentation?.

---

Step by step

Click here for a step by step standard use in french of the configuration LMDZOR_v4.

Choose LMDZ version before extracting

Decide which version of LMDZ you want before extracting the configuration with ./model LMDZOR_v4. For a stable reference version take LMDZ4/branches/LMDZ4_AR5 HEAD version which is the one used for CMIP5 coupled simulations. Or for development issues or to have the latest version take LMDZ5/trunk HEAD version but be aware that it is a not a stable version.

Change eventually in modipsl/util/mod.def, for LMDZ4_AR5 put :

#-C- LMDZOR_v4  LMDZ4/branches/LMDZ4_AR5       HEAD               11 LMDZ       modeles

or for LMDZ5 :

#-C- LMDZOR_v4  LMDZ5/trunk                    HEAD               11 LMDZ       modeles

You can also precise a revision number instead of HEAD to avoid pointing to the latest committed version at the branch indicated just before(LMDZ4/branches/LMDZ4_AR5 or LMDZ5/trunk).

---

CREATE experience for creation of initial state and interpolating of boundary conditions

This experience is used to interpolate boundary conditions(limit.nc and climoz_LMDMZ.nc) and create initial state conditions (start.nc and startphy.nc) for LMDZ using the executable ce0l.e or create_etat0_limit.e. The CREATE experience does not contain a PARAM directory. Instead in CREATE/COMP/lmdz.card the parameter PARAM_DIR points to the directory in experience LMDZOR. PARAM_DIR can be changed to use another directory if wanted.

2 cases for the boundary conditions are predefined :

• using sea-surface temperature(SST) and sea-ice cover(SIC) from climatology AMIP files. This case will create the file limit.nc containg 360 days to be used for running 30days/month. Use cards with suffix _Climat.
• using interannual AMIP files. This case will create the file limit.nc containg 365 days to be used for running with noleap calendar (always 28 days in february). Use cards with suffix _Interannuel.

According to the case you want, copy or link the corresponding .cards. For climatology AMIP :

ln -s config.card_Climat config.card; cd COMP; ln -s lmdz.card_Climat lmdz.card

or for interannual AMIP :

ln -s config.card_Interannuel config.card; cd COMP; ln -s lmdz.card_Interannuel lmdz.card

---

LMDZOR experience for running the gcm

This configuration only contains only one directory for running the gcm.e : LMDZOR. This directory can be used for different type of simulation for example AMIP interannual or climatological run. Or nudged run with regular or zoomed grid. Only the .card changes for different run. The PARAM directory and the .driver are the same. Two possibilities are predefined using the different .cards that are available. To do a climatological simulation, use cards with suffix _Climat, do following link :

cd LMDZOR/COMP; ln -s lmdz.card_Climat lmdz.card; ln -s orchidee.card_Climat orchidee.card

Or to do an AMIP interannual simulation use cards with suffix _AMIP, do following link :

cd LMDZOR/COMP; ln -s lmdz.card_AMIP lmdz.card; ln -s orchidee.card_AMIP orchidee.card

The cards with suffix _Climat correspond to the experience LMDZOR in configuration LMDZ4OR_v3 and the cards with suffix _AMIP correspond to the experience in directory AMIP in LMDZ4OR_v3 configuration.

Parameters in config.card such as JobName, CalendarType, DateBegin, DateEnd must always be considered and eventually modified according to desired experience.

---

Simulation with nudging

For details about nudging options and implementation in LMDZ, see ​LMDZPedia/Nudging , and on LMDZ website this doc in French (dated 2009, some code détails are obsolete): ​guidage_LMDZ.pdf.

Interpolation of nudge files

NOTE : Updated versions of the scripts mentioned below can be downloaded here : ​INTERP_NUDGE.

The files that will be used for the nudging (fr guidage) must be interpolated to the model grid before running the gcm. Scripts to do this are stored in directory CREATE/SCRIPT. To interpolate ERA-Interim use the script ​interp_from_era.ksh. You must first modify the beginning of this script. To interpolate LMDZ model output files use the script ​interp_from_TS.ksh. Both scripts depend on the script ​era2gcm.ksh. These scripts access the archive directory with a use in ferret. Therefore it is not possible to use them from vargas or brodie. These scripts are tested at mercure and ulam. Follow step 1 to 3 :

1) Create file grilles_gcm.nc
The file grilles_gcm.nc is created by default during the CREATE experience if you use revision 1508 of LMDZ5/trunk or later and revision 1333 or later of the configuration LMDZOR_v4. The file grilles_gcm.nc contains the different grids used as target for the interpolation. The file is created with ce0l if the parameter grille_gcm_netcdf=T(default in CREATE). After a normal CREATE experience the file can be found in the archive directory IGCM_OUT/LMDZOR/ELXXXX/ATM/Output/Boundary/. If the grid contains a zoom it is important that the file grilles_gcm.nc is created at the same time as the other boundary conditions and initial state files, using the same .def parameter files describing the grid.

2) Modify and run the script interp_from_era.ksh from a machine that access the ERA files by use in ferret. Script tested at mercure and ulam.

3) Move interpolated files to archive directory.

For running

Follow 3 steps to activate nudging in LMDZ :

1) Set parameter ok_guide=y in lmdz.card. The lmdz.driver will then put ok_guide=y in the guide.def file.

2) Add files containing the nudging variables in lmdz.card (_Climat or _AMIP) in section BoundaryFiles. These files have to be interpolated previously to the model horizontal grid.

3) Most probably you have to modify parameter file guide.def containing specifications for the nudging according to your experience.

---

Different model resolution

Paragraphe to be written...
While waiting, remember that every step is depending on the resolution and zoom : compiling, creation of boudary conditions, interpolation of nudging files, final simulation. If you change the zoom parameters, you do not have to recompile but you have to re-create boudary conditions and re-do interpolation of nudging files.

Using IPSL coupled model output

Creating boundary condition file limit.nc

Do following steps to create boundary conditions using output from a previous LMDZ run.

NB: it is not possible to use version LMDZ4_AR5 of ce0l. Use version LMDZ5/trunk revision 1508 or later. Once the boundary files limit.nc are created they can be used also with earlyer versions of LMDZ such as LMDZ4_AR5.

Step 1
Identify time series for the simulations you want and split them into yearly files. For the SST use the variable tsol_oce and for SIC use the variable pourc_sic. The variables should be at monthly time series format from LMDZ. For example for the coupled IPSLCM5A rcp4.5 simulation, at CCRT use the files :

/dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_tsol_oce.nc
/dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_pourc_sic.nc

Use cdo to split the time series into yearly files with 12 time steps in each file. For exemple :

cdo splityear all_path_to_file_TS_MO_tsol_oce.nc  $WORKDIR/my_new_dir/v3.rcp45.1_tsol_oce_
cdo splityear all_path_to_file_TS_MO_pourc_sic.nc $WORKDIR/my_new_dir/v3.rcp45.1_pourc_sic_

Step 2
The yearly files are now directly usable with the ce0l program revision 1508 LMDZ5/trunk or later. They should be renamed as histmth_sst.nc and histmth_sic.nc before running. The AMIP files must not be in the same run directory, in that case they will be used instead of the model output files. Use the CREATE interannual experience to create the boundary conditions for the interval of years needed. Change in lmdz.card_Interannual AMIP yearly files towards yearly time series :

[BoundaryFiles]
List=   (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sst_360x180_${year}.nc, amipbc_sst_1x1.nc), \
        (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sic_360x180_${year}.nc, amipbc_sic_1x1.nc), \

into

[BoundaryFiles]
List=   (/the/whole/path/to/the/yearly/files_tosl_oce_${year}.nc,  histmth_sst.nc), \
        (/the/whole/path/to/the/yearly/files_pourc_sic_${year}.nc, histmth_sic.nc), \

Modify the period in config.card_Interannual. Don't forget to copy or link the lmdz.card_Interannal into lmdz.card and config.card_Interannual into config.card. Submit the experience.

Creating boundary files containing SST with anomaly

The SST from the coupled model can be used directly as above or corrected with the difference between a reference SST from AMIP and a reference SST from the coupled model output. These two references should be at the same period for example for the years 1979-2005. For example for the simulation RCP 4.5, use SST from historical simulation as reference compare to AMIP SST.

To do this, use the script CREATE/SCRIPT/create_sst_anomly.ksh instead of step 1 above. See the script ​here.

Files already create with these methods

see this page

Use restart files from a coupled simulation to start a forced LMDZ simulation

It is possible to use restart files from a coupled simulation to start a forced simulation but special attention has to be payed to the boundary conditions. It is always necessary to have the same land-ocean mask in the boundary condition file limit.nc and in the restart files. The land-ocean mask in the coupled model is stored in the file o2a.nc. This file has to be present while creating the limit.nc file. In the CREATE directory, add in COMP/lmdz.card, in list "ListNonDel" in section "[BoundaryFiles]" the whole path to the file o2a.nc with the same resolution, for example ${R_INIT}/ATM/IPSLCM5/ORCA2xLMD9695/o2a.nc .

---

Problems running at VARGAS

• In config.card you must change to

  JobRunOptions=''
  

  otherwise you'll have following error message : ERROR: 0031-636 User requested or EOF termination of pm_command
• Vargas have very little memory per processus. For resolution 96x95x39, you need at least 16processus MPI to be able to run.
• The fft filter is not usable for old versions of LMDZ4. Deactivate the filter fft by setting in run.def use_filtre_fft = n. The fft filter is good to use at vargas since revision 1601 at branch LMDZ4_AR5 and revision 1383 at LMDZ5/trunk.
• You should not compile with -O5 at VARGAS. Compile with -O3 or lower optimization. Default compiler xlf 13.1.0.3 is ok with -O3.
• Check memory in your job. For higher resolutions you may need to add threads just to have more memory. You must then compile in mpi mode (default for create_etat0_limit.e.para and gcm.e) and not mix mpi_openmpi. In your job, add for exemple @ parallel_threads = 2. This gives you 2 times more memory per process mpi available and you can multiply data_limit and stack_limit by 2. When adding threads you should remove @ resources = ConsumableCpus(1). For example for resolution 280x280x39 put in your job :

  # @ total_tasks = 32
  # @ parallel_threads = 2
  # @ data_limit  = 6.4gb
  # @ stack_limit = 1.0gb,1.0gb
  ### @ resources = ConsumableCpus(1)
  

• If the executable create_etat0_limit or ce0l exits with following error message you need to update your LMDZ version to revision 1476 at branch LMDZ4_AR5 or higher :

ERROR: 0032-184 MPI was not finalized in routine unknown, task 0

---

How to run LMDZ from the coupled model in forced mode

The LMDZ executable compiled in the coupled configuration can also be used in forced mode (with or without ORCHIDEE). This is possible for versions from LMDZ4_V3_2 and later. Follow steps below to do this keeping the same modipsl without recompiling the model.

1) Extract the configuration LMDZOR_v4 and place it beside the coupled configuration.

>cd modipsl/config
>svn co http://forge.ipsl.jussieu.fr/igcmg/svn/CONFIG/LMDZOR/branches/LMDZOR_v4 LMDZOR_v4
>ls
IPSLCM5A  LMDZOR_v4

2) Create the file .resol
While running an experience the file .resol must exist in config/LMDZOR_v4 directory. Normally this file is created after successful compilation but in this case we do not want to recompile so this file as to be created manually. Look in AA_make to see how this file should look like. For example :

config/LMDZOR_v4> cat .resol 
LMD9695-L39
RESOL_ATM_3D=96x95x39

3) Change name of executable in CREATE experience
The executable have a slightly different name while compiled in the coupled configuration. Change in config.card in section [Executable] to take the correct name of the executable. The syntax is as follow, change my_name_of_exec :

[Executable]
Name=create_etat0_limit.e
#D- For each component, Real name of executable, Name of executable for oasis
ATM= (my_name_of_exec, create_etat0_limit.e)

4) Launch the executable in parallel mode in CREATE experience
The create_etat0_limit can be compiled in sequential or parallel mode. The compilation done in the coupled configuration IPSLCM5A is parallel mode. If you run at an SX8 or SX9 machine, this executable has to be launched with mpirun even though it will only run on one process. To do this, modify in the job as follow :

#    if ( [ X${BATCH_NUM_PROC_TOT} != X ] && [ "${BATCH_NUM_PROC_TOT}" -gt 1 ] ) ; then
        MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND}
        eval MPIRUN_OPTIONS=${config_UserChoices_JobRunOptions}
#    else
#        MPIRUN_COMMAND=${MPIRUN_COMMAND:="time "}
#        MPIRUN_OPTIONS=${MPIRUN_OPTIONS:=""}
#    fi

4) No difference for LMDZOR experience

---

Without ORCHIDEE

In this configuration LMDZ is compiled and executed with ORCHIDEE. Still the LMDZ executable compiled with ORCHIDEE can also be used without ORCHIDEE and instead the land surface bucket model in LMDZ will be used. This choice is controled by the parameter VEGET=y/n, see run section below. But if you don't have acces to the ORCHIDEE sources or for another reason you don't want to compile ORCHIDEE, then you have to edit the main makefile.

Compile without ORCHIDEE

In modipsl/config/LMDZOR_v4/AA_make :

• remove liborchidee from the target you want to compile and create Makefile with ./ins_make. For exemple if you want to compile the resolution 96x95x39, change

  LMD9695-L39 : libioipsl liborchidee lmdz96x95x39 verif
  

  into

  LMD9695-L39 : libioipsl lmdz96x95x39 verif
  

• remove "-v true" from the arguments to makelmdz_fcm and remove the suffix _orch in the name of binaries. The target lmdz in the end of the file becomes then :

lmdz:
        (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e.seq ; )
        (cd ../../modeles/LMDZ; ./makelmdz_fcm -cpp ORCHIDEE_NOOPENMP -d $(RESOL_LMDZ) -parallel mpi -arch $(FCM_ARCH) gcm ; cp bin/gcm_$(RESOL_LMDZ)_phylmd_para.e ../../bin/gcm.e ; )
        (cd ../../modeles/LMDZ; ./makelmdz_fcm -cpp ORCHIDEE_NOOPENMP -d $(RESOL_LMDZ) -parallel mpi -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_para.e ../../bin/create_etat0_limit.e.para ; )

Run without ORCHIDEE

To deactivate ORCHIDEE in the simulation, in LMDZ the only difference is to put VEGET=n in run.def. While submitting using libIGCM also the config.card must be modified. Do following in the LMDZOR experience (no difference for CREATE) :

• config.card : Remove the all lines related to SRF(ORCHIDEE) and SBG(STOMATE) in section [ListOfComponents]

• run.def : Modify in PARAM/run.def to VEGET=n

Using the new physics package in LMDZ

The new physics is still under developpement but there is now a working version called NPv3.0 distributed in LMDZOR_v4 and IPSLCM5B. To use this version you need LMDZ5/trunk revision 1554 or later. Activate this version putting in lmdz.card LMDZ_Physics=NPv3.0, default value is AP for old physics. Then physiq.def_L39_NPv3.0 file in PARAM directory will be used. Only 39 vertical levels is tested and available with this version.

Start a simulation with new physics using restart files from a simulation with old physics at the same resolution (and with same zoom). The restart files must come from a simulation that has run for at least some days. It is not possible to start using the results from create_etat0_limit. If the modele is not stable, activate purmats for the first month to stabilize the modele. Do this by putting !ByPass_hgardfou_mats=y in lmdz.card.

Running LMDZOR_v4 on Mac OS X 10.7 (Lion)

Prerequisites

• Mac OS X 10.7 (Lion).
• XCode 4.2. See App Store.
• gfortran. See : ​http://gcc.gnu.org/wiki/GFortranBinaries#MacOS
• netcdf installed through fink. See : ​http://www.finkproject.org/download/srcdist.php

  bash-3.2$ fink list '*netcdf*'
  Information about 3727 packages read in 1 seconds.
       nco-netcdf                  4.0.8-4              The NetCDF Operators
       netcdf                      3.6.3-4              Interface & lib for array-based data access
       netcdf-bin                  3.6.3-4              Interface & lib for array-based data access
       netcdf-shlibs               3.6.3-4              Interface & lib for array-based data access
   i   netcdf7                     4.1.3-2              Array-based data access, headers and docs
   i   netcdf7-bin                 4.1.3-2              Array-based data access, interface
   i   netcdf7-gfortran            4.1.3-5              Interface & lib for array-based data access
   i   netcdf7-gfortran-shlibs     4.1.3-5              Interface & lib for array-based data access
   i   netcdf7-shlibs              4.1.3-2              Array-based data access, headers and docs
  

  • fink installs netcdf here : /sw/opt/netcdf7/include

Test done : november 2011

Compilation

svn_ano
cd modipsl/util
./model LMDZOR_v4
./ins_make -p I4R4
cd ../config/LMDZOR_v4
## Modif Makefile to add LMD3224-L11-seq target
make LMD3224-L11-seq 2>&1 | tee compile.out

Execution

Adapted from install.sh script.

grid_resolution=32x24x11
bench=bench_lmdz_${grid_resolution}
wget http://www.lmd.jussieu.fr/~lmdz/DistribG95/$bench.tar.gz
tar xzvf $bench.tar
cd BENCH32x24x11
vi run.def  # rm orchidee.def
./bench.sh > bench.out  2>&1
echo '##########################################################'
echo ' Resultat du bench '
echo '##########################################################'
cat ./bench.out

Run a CMIP5 experience

The set up for all CMIP5 simulations have been saved here : ​http://forge.ipsl.jussieu.fr/igcmg/browser/CMIP5

For each simulation the run directory is saved; containing the COMP, PARAM and POST sub directories and the config.card. It is possible to download and use directly or with some modifications. To do so, first extract the configuration LMDZOR_v4 as usual and compile the gcm. Then extract the CMIP5 run directory and do your simulation in that directory. For exemple with simulation v3cf.amip5 :

cd modipsl/util ; ./model LMDZOR_v4 ; ins_make
cd ../config/LMDZOR_v4; gmake
svn co http://forge.ipsl.jussieu.fr/igcmg/svn/CMIP5/amip/v3cf.amip5 .
cd v3cf.amip5   => adapt and do your simulation

Some difficulties

• If you want to run a coupled IPSL-CM5 simluation in forced mode with LMDZOR_v4 you have to add the limit.nc file in lmdz.card. You also have to modify ListOfComponents in the config.card so that only ATM, SRF and SBG are left. And you have to put type_ocean=force dans PARAM/run.def .
• Land-ocean mask : The restart files and the limit.nc file must contain the same land-ocean mask. If you want to restart from a coupled simulation, the limit.nc must have been produced using the o2a.nc file.
• If you want to be sure to have the same model as used in the original simulation, you have to extract the same revision number (modify in mod.def as described above). Be aware that early revision number on branch LMDZ4_AR5 does not run with filtre_fft=y at Vargas(IDRIS). Use the head of LMDZ4_AR5 or put filtre_fft=n in run.def.