LMDZOR_v4 configuration

Josefine Ghattas

LMDZOR_v4 is the current version using LMDZ and ORCHIDEE. It can be used with LMDZ4 production version LMDZ4_AR5 or with one of the latest versions of LMDZ5/trunk. Therefore before extracting the model think about which version of LMDZ you want and correct eventually in modipsl/util/mod.def. Before running this model set up, choose between 2 mode : Interannual AMIP simulation or climatological simulation. No default setup exists. Modifications done by the user in the lmdz.card or orchidee.card are often necessary to adapt to a specific simulation. Modifications in lmdz.driver or orchidee.driver are less often necessary.

---

Choose LMDZ version before extracting

Decide which version of LMDZ you want before extracting the configuration with ./model LMDZOR_v4. For a stable reference version take LMDZ4/branches/LMDZ4_AR5 HEAD version which is the one used for CMIP5 coupled simulations. Or for development issues or to have the latest version take LMDZ5/trunk HEAD version but be aware that it is a not a stable version.

Change eventually in modipsl/util/mod.def, for LMDZ4_AR5 put :

#-C- LMDZOR_v4  LMDZ4/branches/LMDZ4_AR5       HEAD               11 LMDZ       modeles

or for LMDZ5 :

#-C- LMDZOR_v4  LMDZ5/trunk                    HEAD               11 LMDZ       modeles

You can also precise a revision number instead of HEAD to avoid pointing to the latest committed version at the branch indicated just before(LMDZ4/branches/LMDZ4_AR5 or LMDZ5/trunk).

---

CREATE experience for creation of initial state and interpolating of boundary conditions

This experience is used to interpolate boundary conditions(limit.nc and climoz_LMDMZ.nc) and create initial state conditions (start.nc and startphy.nc) for LMDZ using the executable ce0l.e or create_etat0_limit.e. The CREATE experience does not contain a PARAM directory. Instead in CREATE/COMP/lmdz.card the parameter PARAM_DIR points to the directory in experience LMDZOR. PARAM_DIR can be changed to use another directory if wanted.

2 cases for the boundary conditions are predefined : creation using sea-surface temperature(SST) and sea-ice cover(SIC) from climatology AMIP files or from interannual AMIP files. According to the case you want, copy or link the corresponding .cards. For climatology AMIP :

ln -s config.card_Climat config.card; cd COMP; ln -s lmdz.card_Climat lmdz.card

or for interannual AMIP :

ln -s config.card_Interannuel config.card; cd COMP; ln -s lmdz.card_Interannuel lmdz.card

Using IPSL coupled model output

Do following steps to create boundary conditions using output from a previous LMDZ run.

NB: it is not possible to use version LMDZ4_AR5 of ce0l. Use version LMDZ5/trunk revision 1508 or later. Once the boundary files limit.nc are created they can be used also with earlyer versions of LMDZ such as LMDZ4_AR5.

Step 1
Identify time series for the simulations you want and split them into yearly files. For the SST use the variable tsol_oce and for SIC use the variable pourc_sic. The variables should be at monthly time series format from LMDZ. For example for the coupled IPSLCM5A rcp4.5 simulation, at CCRT use the files :

/dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_tsol_oce.nc
/dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_pourc_sic.nc

Use cdo to split the time series into yearly files with 12 time steps in each file. For exemple :

cdo splityear TS_MO_tsol_oce  $WORKDIR/my_new_dir/v3.rcp45.1_tsol_oce_
cdo splityear TS_MO_pourc_sic $WORKDIR/my_new_dir/v3.rcp45.1_pourc_sic_

Step 2
The yearly files are now directly usable with the ce0l program revision 1508 LMDZ5/trunk or later. They should be renamed as histmth_sst.nc and histmth_sic.nc before running. The AMIP files must not be in the same run directory, in that case they will be used instead of the model output files. Use the CREATE interannual experience to create the boundary conditions for the interval of years needed. Change in lmdz.card_Interannual AMIP yearly files towards yearly time series :

[BoundaryFiles]
List=   (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sst_360x180_${year}.nc, amipbc_sst_1x1.nc), \
        (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sic_360x180_${year}.nc, amipbc_sic_1x1.nc), \

into

[BoundaryFiles]
List=   (/the/whole/path/to/the/yearly/files_tosl_oce_${year}.nc,  histmth_sst.nc), \
        (/the/whole/path/to/the/yearly/files_pourc_sic_${year}.nc, histmth_sic.nc), \

Modify the period in config.card_Interannual. Don't forget to copy or link the lmdz.card_Interannal into lmdz.card and config.card_Interannual into config.card. Submit the experience.

SST anomaly

The SST from the coupled model can be used directly as above or corrected with the difference between a reference SST from AMIP and a reference SST from the coupled model output. These two references should be at the same period for example for the years 1979-2005. For example for the simulation RCP 4.5, use SST from historical simulation as reference compare to AMIP SST.

To do this, use the script CREATE/SCRIPT/create_sst_anomly.ksh instead of step 1 above. See the script ​here.

Interpolation of nudge files

The files that will be used for the nudging (fr guidage) must be interpolated to the model grid before running the gcm. Scripts to do this are stored in directory CREATE/SCRIPT. To interpolate ERA-Interim use the script ​interp_from_era.ksh. You must first modify the beginning of this script. To interpolate LMDZ model output files use the script ​interp_from_TS.ksh. Both scripts depend on the script ​era2gcm.ksh. These script access the archive directory with a use in ferret. Therefore it is not possible to use them from vargas or brodie. These scripts are tested at mercure and ulam. Follow step 1 to 3 :

1) Create file grilles_gcm.nc
The file grilles_gcm.nc is created by default during the CREATE experience if you use revision 1508 of LMDZ5/trunk or later and revision 1333 or later of the configuration LMDZOR_v4. The file grilles_gcm.nc contains the different grids used as target for the interpolation. The file is created with ce0l if the parameter grille_gcm_netcdf=T(default in CREATE). After a normal CREATE experience the file can be found in the archive directory IGCM_OUT/LMDZOR/ELXXXX/ATM/Output/Boundary/. If the grid contains a zoom it is important that the file grilles_gcm.nc is created at the same time as the other boundary conditions and initial state files, using the same .def parameter files describing the grid.

2) Modify and run the script interp_from_era.ksh from a machine that access the ERA files by use in ferret. Script tested at mercure and ulam.

3) Move interpolated files to archive directory.

---

LMDZOR experience for running the gcm

This configuration only contains only one directory for running the gcm.e : LMDZOR. This directory can be used for different type of simulation for example AMIP interannual or climatological run. Or nudged run with regular or zoomed grid. Only the .card changes for different run. The PARAM directory and the .driver are the same. Two possibilities are predefined using the different .cards that are available. To do a climatological simulation, use cards with suffix _Climat, do following link :

cd LMDZOR/COMP; ln -s lmdz.card_Climat lmdz.card; ln -s orchidee.card_Climat orchidee.card

Or to do an AMIP interannual simulation use cards with suffix _AMIP, do following link :

cd LMDZOR/COMP; ln -s lmdz.card_AMIP lmdz.card; ln -s orchidee.card_AMIP orchidee.card

The cards with suffix _Climat correspond to the experience LMDZOR in configuration LMDZ4OR_v3 and the cards with suffix _AMIP correspond to the experience in directory AMIP in LMDZ4OR_v3 configuration.

Parameters in config.card such as JobName, CalendarType, DateBegin, DateEnd must always be considered and eventually modified according to desired experience.

Nudged run

Follow 3 steps to activate nudging in LMDZ :

1) Set parameter ok_guide=y in lmdz.card. The lmdz.driver will then put ok_guide=y in the guide.def file.

2) Add files containing the nudging variables in lmdz.card (_Climat or _AMIP) in section BoundaryFiles. These files have to be interpolated previously to the model horizontal grid.

3) Most probably you have to modify parameter file guide.def containing specifications for the nudging according to your experience.

---

Problems running at VARGAS

• In config.card you must change to

  JobRunOptions=''
  

  otherwise you'll have following error message : ERROR: 0031-636 User requested or EOF termination of pm_command
• You must deactivate the filter fft if using branch LMDZ4_AR5 or earlier version of LMDZ. In run.def set use_filtre_fft = n.
• You should not compile with -O5 at VARGAS. Compile with -O3 or lower optimization. Default compiler xlf 13.1.0.3 is ok with -O3.
• Check memory in your job. For higher resolutions you may need to add threads just to have more memory. You must then compile in mpi mode (default for create_etat0_limit.e.para and gcm.e) and not mix mpi_openmpi. In your job, add for exemple @ parallel_threads = 2. This gives you 2 times more memory per process mpi available and you can multiply data_limit and stack_limit by 2. When adding threads you should remove @ resources = ConsumableCpus(1). For example for resolution 280x280x39 put in your job :

  # @ total_tasks = 32
  # @ environment = "BATCH_NUM_PROC_TOT=32"
  # @ parallel_threads = 2
  # @ data_limit  = 6.4gb
  # @ stack_limit = 1.0gb,1.0gb
  ### @ resources = ConsumableCpus(1)
  

• If the executable create_etat0_limit or ce0l exits with following error message you need to update your LMDZ version to revision 1476 at branch LMDZ4_AR5 or higher :

ERROR: 0032-184 MPI was not finalized in routine unknown, task 0

---

How to run LMDZ from the coupled model in forced mode

The LMDZ executable compiled in the coupled configuration can also be used in forced mode (with or without ORCHIDEE). This is possible for versions from LMDZ4_V3_2 and later. Follow steps below to do this keeping the same modipsl without recompiling the model.

1) Extract the configuration LMDZOR_v4 and place it beside the coupled configuration.

>cd modipsl/config
>svn co http://forge.ipsl.jussieu.fr/igcmg/svn/CONFIG/LMDZOR/branches/LMDZOR_v4 LMDZOR_v4
>ls
IPSLCM5A  LMDZOR_v4

2) Create the file .resol
While running an experience the file .resol must exist in config/LMDZOR_v4 directory. Normally this file is created after successful compilation but in this case we do not want to recompile so this file as to be created manually. Look in AA_make to see how this file should look like. For example :

config/LMDZOR_v4> cat .resol 
LMD9695-L39
RESOL_ATM_3D=96x95x39

3) Change name of executable in CREATE experience
The executable have a slightly different name while compiled in the coupled configuration. Change in config.card in section [Executable] to take the correct name of the executable. The syntax is as follow, change my_name_of_exec :

[Executable]
Name=create_etat0_limit.e
#D- For each component, Real name of executable, Name of executable for oasis
ATM= (my_name_of_exec, create_etat0_limit.e)

4) Launch the executable in parallel mode in CREATE experience
The create_etat0_limit can be compiled in sequential or parallel mode. The compilation done in the coupled configuration IPSLCM5A is parallel mode. If you run at an SX8 or SX9 machine, this executable has to be launched with mpirun even though it will only run on one process. To do this, modify in the job as follow :

#    if ( [ X${BATCH_NUM_PROC_TOT} != X ] && [ "${BATCH_NUM_PROC_TOT}" -gt 1 ] ) ; then
        MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND}
        eval MPIRUN_OPTIONS=${config_UserChoices_JobRunOptions}
#    else
#        MPIRUN_COMMAND=${MPIRUN_COMMAND:="time "}
#        MPIRUN_OPTIONS=${MPIRUN_OPTIONS:=""}
#    fi

4) No difference for LMDZOR experience

---

How to deactivate ORCHIDEE

The LMDZ executable can run without ORCHIDEE even if it is compiled with ORCHIDEE. Instead the land surface bucket model in LMDZ will be used. To deactivate ORCHIDEE in LMDZ the only difference is to put VEGET=n in run.def. While submitting using libIGCM also the config.card must be modified. Do following in the LMDZOR experience (no difference for CREATE) :

• config.card : Remove the whole section [SRF] and [SBG] and all lines related to SRF(ORCHIDEE) and SBG(STOMATE) in section [ListOfComponents] and [Executable]. Now the .cards and .drivers related to ORCHIDEE and STOMATE will not be used.

• run.def : Modify in PARAM/run.def to VEGET=n