If you've never really used LMDZ before, coupling is a scary event, even if you know ORCHIDEE quite well. Thankfully, there are some tips to make things a bit easier.

Tip 1) check coupling

If the job crashes immediately (i.e., within a day), it's good to check the coupling between the two models, which variables are passed and what their values are. This can be found in intersurf_gathered, in src_sechiba/intersurf.f90. If you changed the flag check_INPUTS in this file to .TRUE. and recompile, it will create a whole bunch of useful files in the run directory when run, of the format W*.nc (Walb_nir.nc, Wfluxlat.nc, etc., one for all the variables passed to ORCHIDEE and passed from ORCHIDEE). You can look at these with the standard NetCDF tools for each step to see if the values are unusual. In order to determine what is "usual", I do a one day run on the same resolution with the trunk and all the options I'm interested in (11-layer hydrology, STOMATE, new physics, etc.) and enable the check_INPUTS flag. This gives me an identical set of files to compare to.

Question: where is the run directory? It is not the submit directory (config/, where the job is submitted from), and it is not the archive directory (IGCM_OUT/, where the output files are stored). It is a temporary directory where all the files are copied to and execution takes place. If you look in the Script_* output file in the submit directory of a job, you can find a line which looks something like this (on Curie)

IGCM_sys_Cd : /ccc/scratch/cont003/dsm/p529grat/RUN_DIR/1213660_24665/DOFOCO.24665

This is the run directory. It is deleted if the run completes successfully, but not deleted if a crash happens. You can also find this directory based on the job submission ID.

$SCRATCHDIR/RUN_DIR/number_id_job_......

number_id_job is the number given by the system when you submit the job

For example,

[p86cozic@curie70 loig.default]$ ccc_msub Job_loig.default
Submitted Batch Session 1213759

This RUN_DIR will be : $SCRATCHDIR/RUN_DIR/1213759_*

The origin of the second set of numbers (the *) is unclear, but the job ID is unique, so there should only be one directory which starts with that number.

Tip 2) Turn on debug flags for ORCHIDEE and LMDZ

ORCHIDEE

The compiling of ORCHIDEE has changed so depending on when you extracted the model you should choose the appropriate method.

Find out which method is used in your version by opening the file modipsl/config/LMDZOR_v5.2/AA_make. If it contains

LMD9695-L39 : libioipsl liborchideefcm lmdz96x95x39 verif

you have the new method based on the script makeorchidee_fcm. For this case, add at the end of the line with makeorchidee_fcm -debug to the following :

liborchideefcm : ../../modeles/ORCHIDEE/
        (cd ../../modeles/ORCHIDEE/ ; ./makeorchidee_fcm -parallel mpi -arch $(FCM_ARCH) -driver -j 8 -debug )

If instead the file modipsl/config/LMDZOR_v5.2/AA_make contains

LMD9695-L39 : libioipsl liborchidee lmdz96x95x39 verif

you have the old method with standard makefiles based on AA_make.gdef. Change in modipsl/util/AA_make.gdef the line corresponding to F_O for your machine.

For example at curie set:

#-Q- curie  F_O = -DCPP_PARA -p -g -traceback -fp-stack-check -ftrapuv -check bounds $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR)

For example at ada set: #-Q- ada F_O = -DCPP_PARA -p -g -traceback -fp-stack-check -ftrapuv -check bounds $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR)

Note that these lines are proposed in AA_make.gdef in commented form using ####-Q-...

LMDZ
For LMDZ, add -debug on the 2 lines containing makelmdz_fcm in the same file modipsl/config/LMDZOR_v5.2/AA_make. You should have the following:

lmdz: ../../modeles/LMDZ
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -debug -parallel mpi -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/create_etat0_limit.e ; )
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -debug -parallel mpi -arch $(FCM_ARCH) gcm ;  cp bin/gcm_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/gcm.e ; )

Recreate makefiles and compile
You must now recreate the Makefile, clean and recompile. Do as follow:

cd modipsl/config/LMDZOR_v5.2
../../util/ins_make
gmake clean
gmake

This will recompile both ORCHIDEE and LMDZ with the new options you set.

Note: The debug flags, as well as all compile options, that are used for ORCHIDEE compiled with makeorechidee_fcm are found in modeles/ORCHIDEE/arch/arch-....fcm and for LMDZ in modeles/LMDZ/arch/arch-....fcm.