Version 24 (modified by jgipsl, 13 years ago) (diff) |
---|
- LMDZOR_v4 configuration
- Step by step
- Choose LMDZ version before extracting
- CREATE experience for creation of initial state and interpolating of …
- LMDZOR experience for running the gcm
- Simulation with nudging
- Different model resolution
- Using IPSL coupled model output
- Problems running at VARGAS
- How to run LMDZ from the coupled model in forced mode
- Without ORCHIDEE
- Using the new physics package in LMDZ
- Running LMDZOR_v4 on Mac OS X 10.7 (Lion)
LMDZOR_v4 configuration
Josefine Ghattas
LMDZOR_v4 is the current version using LMDZ and ORCHIDEE. It can be used with LMDZ4 production version LMDZ4_AR5 or with one of the latest versions of LMDZ5/trunk. Therefore before extracting the model think about which version of LMDZ you want and correct eventually in modipsl/util/mod.def.
Before running this model set up, choose between 2 mode : Interannual AMIP simulation or climatological simulation. No default setup exists. Modifications done by the user in the lmdz.card or orchidee.card are often necessary to adapt to a specific simulation. Modifications in lmdz.driver or orchidee.driver are less often necessary.
In this page you will find information specific to the configuration LMDZOR_v4. This page does not intend to explain how to use LMDZOR_v4 configuration! To learn more about using the configurations in modipsl see => ModipslBeginner.
Step by step
Click here for a step by step standard use in french of the configuration LMDZOR_v4.
Choose LMDZ version before extracting
Decide which version of LMDZ you want before extracting the configuration with ./model LMDZOR_v4. For a stable reference version take LMDZ4/branches/LMDZ4_AR5 HEAD version which is the one used for CMIP5 coupled simulations. Or for development issues or to have the latest version take LMDZ5/trunk HEAD version but be aware that it is a not a stable version.
Change eventually in modipsl/util/mod.def, for LMDZ4_AR5 put :
#-C- LMDZOR_v4 LMDZ4/branches/LMDZ4_AR5 HEAD 11 LMDZ modeles
or for LMDZ5 :
#-C- LMDZOR_v4 LMDZ5/trunk HEAD 11 LMDZ modeles
You can also precise a revision number instead of HEAD to avoid pointing to the latest committed version at the branch indicated just before(LMDZ4/branches/LMDZ4_AR5 or LMDZ5/trunk).
CREATE experience for creation of initial state and interpolating of boundary conditions
This experience is used to interpolate boundary conditions(limit.nc and climoz_LMDMZ.nc) and create initial state conditions (start.nc and startphy.nc) for LMDZ using the executable ce0l.e or create_etat0_limit.e. The CREATE experience does not contain a PARAM directory. Instead in CREATE/COMP/lmdz.card the parameter PARAM_DIR points to the directory in experience LMDZOR. PARAM_DIR can be changed to use another directory if wanted.
2 cases for the boundary conditions are predefined : creation using sea-surface temperature(SST) and sea-ice cover(SIC) from climatology AMIP files or from interannual AMIP files. According to the case you want, copy or link the corresponding .cards. For climatology AMIP :
ln -s config.card_Climat config.card; cd COMP; ln -s lmdz.card_Climat lmdz.card
or for interannual AMIP :
ln -s config.card_Interannuel config.card; cd COMP; ln -s lmdz.card_Interannuel lmdz.card
LMDZOR experience for running the gcm
This configuration only contains only one directory for running the gcm.e : LMDZOR. This directory can be used for different type of simulation for example AMIP interannual or climatological run. Or nudged run with regular or zoomed grid. Only the .card changes for different run. The PARAM directory and the .driver are the same. Two possibilities are predefined using the different .cards that are available. To do a climatological simulation, use cards with suffix _Climat, do following link :
cd LMDZOR/COMP; ln -s lmdz.card_Climat lmdz.card; ln -s orchidee.card_Climat orchidee.card
Or to do an AMIP interannual simulation use cards with suffix _AMIP, do following link :
cd LMDZOR/COMP; ln -s lmdz.card_AMIP lmdz.card; ln -s orchidee.card_AMIP orchidee.card
The cards with suffix _Climat correspond to the experience LMDZOR in configuration LMDZ4OR_v3 and the cards with suffix _AMIP correspond to the experience in directory AMIP in LMDZ4OR_v3 configuration.
Parameters in config.card such as JobName, CalendarType, DateBegin, DateEnd must always be considered and eventually modified according to desired experience.
Simulation with nudging
For details about nudging options and implementaion in LMDZ, see a description in french at the LMDZ web-site, choose the second joint file called "Guidage de LMDZ".
Interpolation of nudge files
The files that will be used for the nudging (fr guidage) must be interpolated to the model grid before running the gcm. Scripts to do this are stored in directory CREATE/SCRIPT. To interpolate ERA-Interim use the script interp_from_era.ksh. You must first modify the beginning of this script. To interpolate LMDZ model output files use the script interp_from_TS.ksh. Both scripts depend on the script era2gcm.ksh. These script access the archive directory with a use in ferret. Therefore it is not possible to use them from vargas or brodie. These scripts are tested at mercure and ulam. Follow step 1 to 3 :
1) Create file grilles_gcm.nc
The file grilles_gcm.nc is created by default during the CREATE experience if you use revision 1508 of LMDZ5/trunk or later and revision 1333 or later of the configuration LMDZOR_v4.
The file grilles_gcm.nc contains the different grids used as target for the interpolation. The file is created with ce0l if the parameter grille_gcm_netcdf=T(default in CREATE). After a normal CREATE experience the file can be found in the archive directory IGCM_OUT/LMDZOR/ELXXXX/ATM/Output/Boundary/. If the grid contains a zoom it is important that the file grilles_gcm.nc is created at the same time as the other boundary conditions and initial state files, using the same .def parameter files describing the grid.
2) Modify and run the script interp_from_era.ksh from a machine that access the ERA files by use in ferret. Script tested at mercure and ulam.
3) Move interpolated files to archive directory.
For running
Follow 3 steps to activate nudging in LMDZ :
1) Set parameter ok_guide=y in lmdz.card. The lmdz.driver will then put ok_guide=y in the guide.def file.
2) Add files containing the nudging variables in lmdz.card (_Climat or _AMIP) in section BoundaryFiles. These files have to be interpolated previously to the model horizontal grid.
3) Most probably you have to modify parameter file guide.def containing specifications for the nudging according to your experience.
Different model resolution
Paragraphe to be written...
While waiting, remember that every step is depending on the resolution and zoom : compiling, creation of boudary conditions, interpolation of nudging files, final simulation. If you change the zoom parameters, you do not have to recompile but you have to re-create boudary conditions and re-do interpolation of nudging files.
Using IPSL coupled model output
Do following steps to create boundary conditions using output from a previous LMDZ run.
NB: it is not possible to use version LMDZ4_AR5 of ce0l. Use version LMDZ5/trunk revision 1508 or later. Once the boundary files limit.nc are created they can be used also with earlyer versions of LMDZ such as LMDZ4_AR5.
Step 1
Identify time series for the simulations you want and split them into yearly files. For the SST use the variable tsol_oce and for SIC use the variable pourc_sic. The variables should be at monthly time series format from LMDZ. For example for the coupled IPSLCM5A rcp4.5 simulation, at CCRT use the files :
/dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_tsol_oce.nc /dmnfs/cont003/p86denv/IGCM_OUT/IPSLCM5A/PROD/rcp45/v3.rcp45.1/ATM/Analyse/TS_MO/v3.rcp45.1_20060101_21251231_1M_pourc_sic.nc
Use cdo to split the time series into yearly files with 12 time steps in each file. For exemple :
cdo splityear all_path_to_file_TS_MO_tsol_oce.nc $WORKDIR/my_new_dir/v3.rcp45.1_tsol_oce_ cdo splityear all_path_to_file_TS_MO_pourc_sic.nc $WORKDIR/my_new_dir/v3.rcp45.1_pourc_sic_
Step 2
The yearly files are now directly usable with the ce0l program revision 1508 LMDZ5/trunk or later. They should be renamed as histmth_sst.nc and histmth_sic.nc before running. The AMIP files must not be in the same run directory, in that case they will be used instead of the model output files. Use the CREATE interannual experience to create the boundary conditions for the interval of years needed. Change in lmdz.card_Interannual AMIP yearly files towards yearly time series :
[BoundaryFiles] List= (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sst_360x180_${year}.nc, amipbc_sst_1x1.nc), \ (${R_INIT}/ATM/${config_UserChoices_TagName}/AMIP/amipbc_sic_360x180_${year}.nc, amipbc_sic_1x1.nc), \
into
[BoundaryFiles] List= (/the/whole/path/to/the/yearly/files_tosl_oce_${year}.nc, histmth_sst.nc), \ (/the/whole/path/to/the/yearly/files_pourc_sic_${year}.nc, histmth_sic.nc), \
Modify the period in config.card_Interannual. Don't forget to copy or link the lmdz.card_Interannal into lmdz.card and config.card_Interannual into config.card. Submit the experience.
Creating boundary files containing SST with anomaly
The SST from the coupled model can be used directly as above or corrected with the difference between a reference SST from AMIP and a reference SST from the coupled model output. These two references should be at the same period for example for the years 1979-2005. For example for the simulation RCP 4.5, use SST from historical simulation as reference compare to AMIP SST.
To do this, use the script CREATE/SCRIPT/create_sst_anomly.ksh instead of step 1 above. See the script here.
Problems running at VARGAS
- In config.card you must change to
JobRunOptions=''
otherwise you'll have following error message : ERROR: 0031-636 User requested or EOF termination of pm_command - Vargas have very little memory per processus. For resolution 96x95x39, you need at least 16processus MPI to be able to run.
- You must deactivate the filter fft if using branch LMDZ4_AR5 or earlier version of LMDZ. In run.def set use_filtre_fft = n.
- You should not compile with -O5 at VARGAS. Compile with -O3 or lower optimization. Default compiler xlf 13.1.0.3 is ok with -O3.
- Check memory in your job. For higher resolutions you may need to add threads just to have more memory. You must then compile in mpi mode (default for create_etat0_limit.e.para and gcm.e) and not mix mpi_openmpi. In your job, add for exemple @ parallel_threads = 2. This gives you 2 times more memory per process mpi available and you can multiply data_limit and stack_limit by 2. When adding threads you should remove @ resources = ConsumableCpus(1). For example for resolution 280x280x39 put in your job :
# @ total_tasks = 32 # @ parallel_threads = 2 # @ data_limit = 6.4gb # @ stack_limit = 1.0gb,1.0gb ### @ resources = ConsumableCpus(1)
- If the executable create_etat0_limit or ce0l exits with following error message you need to update your LMDZ version to revision 1476 at branch LMDZ4_AR5 or higher :
ERROR: 0032-184 MPI was not finalized in routine unknown, task 0
How to run LMDZ from the coupled model in forced mode
The LMDZ executable compiled in the coupled configuration can also be used in forced mode (with or without ORCHIDEE). This is possible for versions from LMDZ4_V3_2 and later. Follow steps below to do this keeping the same modipsl without recompiling the model.
1) Extract the configuration LMDZOR_v4 and place it beside the coupled configuration.
>cd modipsl/config >svn co http://forge.ipsl.jussieu.fr/igcmg/svn/CONFIG/LMDZOR/branches/LMDZOR_v4 LMDZOR_v4 >ls IPSLCM5A LMDZOR_v4
2) Create the file .resol
While running an experience the file .resol must exist in config/LMDZOR_v4 directory. Normally this file is created after successful compilation but in this case we do not want to recompile so this file as to be created manually. Look in AA_make to see how this file should look like. For example :
config/LMDZOR_v4> cat .resol LMD9695-L39 RESOL_ATM_3D=96x95x39
3) Change name of executable in CREATE experience
The executable have a slightly different name while compiled in the coupled configuration. Change in config.card in section [Executable] to take the correct name of the executable. The syntax is as follow, change my_name_of_exec :
[Executable] Name=create_etat0_limit.e #D- For each component, Real name of executable, Name of executable for oasis ATM= (my_name_of_exec, create_etat0_limit.e)
4) Launch the executable in parallel mode in CREATE experience
The create_etat0_limit can be compiled in sequential or parallel mode. The compilation done in the coupled configuration IPSLCM5A is parallel mode. If you run at an SX8 or SX9 machine, this executable has to be launched with mpirun even though it will only run on one process. To do this, modify in the job as follow :
# if ( [ X${BATCH_NUM_PROC_TOT} != X ] && [ "${BATCH_NUM_PROC_TOT}" -gt 1 ] ) ; then MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND} eval MPIRUN_OPTIONS=${config_UserChoices_JobRunOptions} # else # MPIRUN_COMMAND=${MPIRUN_COMMAND:="time "} # MPIRUN_OPTIONS=${MPIRUN_OPTIONS:=""} # fi
4) No difference for LMDZOR experience
Without ORCHIDEE
In this configuration LMDZ is compiled and executed with ORCHIDEE. Still the LMDZ executable compiled with ORCHIDEE can also be used without ORCHIDEE and instead the land surface bucket model in LMDZ will be used. This choice is controled by the parameter VEGET=y/n, see run section below. But if you don't have acces to the ORCHIDEE sources or for another reason you don't want to compile ORCHIDEE, then you have to edit the main makefile.
Compile without ORCHIDEE
In modipsl/config/LMDZOR_v4/AA_make :
- remove liborchidee from the target you want to compile and create Makefile with ./ins_make. For exemple if you want to compile the resolution 96x95x39, change
LMD9695-L39 : libioipsl liborchidee lmdz96x95x39 verif
intoLMD9695-L39 : libioipsl lmdz96x95x39 verif
- remove "-v true" from the arguments to makelmdz_fcm and remove the suffix _orch in the name of binaries. The target lmdz in the end of the file becomes then :
lmdz: (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e.seq ; ) (cd ../../modeles/LMDZ; ./makelmdz_fcm -cpp ORCHIDEE_NOOPENMP -d $(RESOL_LMDZ) -parallel mpi -arch $(FCM_ARCH) gcm ; cp bin/gcm_$(RESOL_LMDZ)_phylmd_para.e ../../bin/gcm.e ; ) (cd ../../modeles/LMDZ; ./makelmdz_fcm -cpp ORCHIDEE_NOOPENMP -d $(RESOL_LMDZ) -parallel mpi -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_para.e ../../bin/create_etat0_limit.e.para ; )
Run without ORCHIDEE
To deactivate ORCHIDEE in the simulation, in LMDZ the only difference is to put VEGET=n in run.def. While submitting using libIGCM also the config.card must be modified. Do following in the LMDZOR experience (no difference for CREATE) :
- config.card : Remove the all lines related to SRF(ORCHIDEE) and SBG(STOMATE) in section [ListOfComponents]
- run.def : Modify in PARAM/run.def to VEGET=n
Using the new physics package in LMDZ
The new physics is still under developpement but there is now a working version called NPv3.0 distributed in LMDZOR_v4 and IPSLCM5B. To use this version you need LMDZ5/trunk revision 1554 or later. Activate this version putting in lmdz.card LMDZ_Physics=NPv3.0, default value is AP for old physics. Then physiq.def_L39_NPv3.0 file in PARAM directory will be used. Only 39 vertical levels is tested and available with this version.
Start a simulation with new physics using restart files from a simulation with old physics at the same resolution (and with same zoom). The restart files must come from a simulation that has run for at least some days. It is not possible to start using the results from create_etat0_limit. If the modele is not stable, activate purmats for the first month to stabilize the modele. Do this by putting !ByPass_hgardfou_mats=y in lmdz.card.
Running LMDZOR_v4 on Mac OS X 10.7 (Lion)
Prerequisites
- Mac OS X 10.7 (Lion).
- XCode 4.2. See App Store.
- gfortran. See : http://gcc.gnu.org/wiki/GFortranBinaries#MacOS
- netcdf installed through fink. See : http://www.finkproject.org/download/srcdist.php
bash-3.2$ fink list '*netcdf*' Information about 3727 packages read in 1 seconds. nco-netcdf 4.0.8-4 The NetCDF Operators netcdf 3.6.3-4 Interface & lib for array-based data access netcdf-bin 3.6.3-4 Interface & lib for array-based data access netcdf-shlibs 3.6.3-4 Interface & lib for array-based data access i netcdf7 4.1.3-2 Array-based data access, headers and docs i netcdf7-bin 4.1.3-2 Array-based data access, interface i netcdf7-gfortran 4.1.3-5 Interface & lib for array-based data access i netcdf7-gfortran-shlibs 4.1.3-5 Interface & lib for array-based data access i netcdf7-shlibs 4.1.3-2 Array-based data access, headers and docs
- fink installs netcdf here : /sw/opt/netcdf7/include
Test done : november 2011
Compilation
svn_ano cd modipsl/util ./model LMDZOR_v4 ./ins_make -p I4R4 cd ../config/LMDZOR_v4 ## Modif Makefile to add LMD3224-L11-seq target make LMD3224-L11-seq 2>&1 | tee compile.out
Execution
Adapted from install.sh script.
grid_resolution=32x24x11 bench=bench_lmdz_${grid_resolution} wget http://www.lmd.jussieu.fr/~lmdz/DistribG95/$bench.tar.gz tar xzvf $bench.tar cd BENCH32x24x11 vi run.def # rm orchidee.def ./bench.sh > bench.out 2>&1 echo '##########################################################' echo ' Resultat du bench ' echo '##########################################################' cat ./bench.out