{{{
#!html
Working on the Irene machine
}}}
----
[[PageOutline(1-3,Table of contents,,numbered)]]
Update 28/06/18
[[NoteBox(warn, Documentation under development , 600px)]]
# On-line users manual #
* https://www-tgcc.ccc.cea.fr/docs/irene.info.pdf (you will need a TGCC login and password)
# Job manager commands #
* {{{ccc_msub job}}} -> submit a job
* {{{ccc_mdel ID}}} -> kill the job with the specified ID number
* {{{ccc_mstat -u login}}} -> display all jobs submitted by login, add {{{-f}}} to see full job name
* {{{ccc_mpp}}} -> display all jobs submitted on the machine. {{{ ccc_mpp -n }}} to avoid colors.
* {{{ ccc_mpp -u $(whoami)}}} ->display your jobs.
# Suggested environment #
Before working on Irene you need to prepare your environment. This is important to do before compilation to ensure the use of same modules as done by libIGCM running environment. We propose you 2 files which you can copy from the home igcmg. The first one called '''bashrc''' will source the second called '''bashrc_irene'''. We suggest that you copy both files to your home, rename them by adding a dot as prefix and modify bashrc so that your bashrc_irene will be sourced instead of the one in igcmg home. Later on you can add personal settings in your .bashrc_irene. Do as follow:
{{{
cd
cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc .bashrc
cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc_irene .bashrc_irene
# Edit .bashrc and modify the path to source your own .bashrc_irene instead of the one in igcmg home.
# After re-connexion or source of .bashrc_irene, check your loaded modules:
module list
Currently Loaded Modulefiles:
1) ccc 10) mkl/17.0.6.256(default) 19) flavor/hdf5/parallel
2) datadir/own(default) 11) flavor/buildcompiler/intel/17(default) 20) netcdf-c/4.6.0(default)
3) dfldatadir/own(default) 12) intel/17.0.6.256(default) 21) netcdf-fortran/4.4.4(default)
4) licsrv/intel 13) hwloc/1.11.3(default) 22) hdf5/1.8.20(default)
5) c++/intel/17.0.6.256(default) 14) feature/openmpi/mpi_compiler/intel(default) 23) feature/bridge/heterogenous_mpmd
6) c/intel/17.0.6.256(default) 15) feature/openmpi/net/mxm(default) 24) nco/4.6.0(default)
7) fortran/intel/17.0.6.256(default) 16) .tuning/openmpi/2.0(default) 25) cdo/1.7.2rc6(default)
8) feature/mkl/lp64 17) flavor/buildmpi/openmpi/2.0 26) ghostscript/9.19(default)
9) feature/mkl/sequential 18) mpi/openmpi/2.0.2 27) ferret/7.2(default)
}}}
The modules are specified in the file /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene which is sourced in bashrc_irene. The same file env_irene is sourced in libIGCM.
--> Be careful this environment can be update during next weeks according to TGCC recommendations
# File system #
On Irene you have a home, a storedir, a workdir, a scratchdir by project. For example if you are working with project gen2201 and gen2212 you will have all these directories
{{{
/ccc/store/cont003/gen2201/login
/ccc/store/cont003/gen2212/login
/ccc/work/cont003/gen2201/login
/ccc/work/cont003/gen2212/login
/ccc/scratch/cont003/gen2201/login
/ccc/scratch/cont003/gen2212/login
/ccc/cont003/home/gen2201/login
/ccc/cont003/home/gen2212/login
}}}
IMPORTANT : Check that you have access to above directories and contact TGCC hotline if it is not the case.
# Required on 2018/7/2 #
Before the stabilisation of Irene environment, you have to use this command before using libIGCM :
{{{
irene > module swith dfldatadir dfldatadir/project_used_for_storage
}}}
# Check your directories #
Before start working you need to know which project you are and load corresponding environment. For example if you will work on the project gen2201, do following (you can the command into your .bashrc_irene):
{{{
module load datadir/gen2201
}}}
By loading a datadir you will have new environment variables to access working directories:
{{{
GEN2201_ALL_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/gen2201
GEN2201_CCCWORKDIR=/ccc/work/cont003/gen2201/login
GEN2201_ALL_HOME=/ccc/cont003/home/gen2201/gen2201
GEN2201_CCCSTOREDIR=/ccc/store/cont003/gen2201/login
GEN2201_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/login
GEN2201_ALL_CCCWORKDIR=/ccc/work/cont003/gen2201/gen2201
GEN2201_HOME=/ccc/cont003/home/gen2201/login
GEN2201_ALL_CCCSTOREDIR=/ccc/store/cont003/gen2201/gen2201
}}}
[[NoteBox(warn, If you previously worked at curie and your directories were in /dsm/ you will now find your data in a specific new project file system "dsmipsl". We recommend to copy your data in your genci project file system , 600px)]]
* Computing nodes: the nodes of partition skylake have 48 cores each, which is 3 times more than the computing nodes from the standard partition of Curie;
* File system access MUST be explicit: from the login nodes, you will see the WORK, SCRATCH, STORE spaces as you probably are used to. However, when submitting any job through ccc_msub or ccc_mprun, you must specify -m work, -m scratch, -m store, or combine them like in -m work,scratch; this constraint has the advantage that your jobs won't be suspended if a file system you don't need becomes unavailable;
* Compute nodes are diskless, meaning that /tmp is not hosted on a local hard drive anymore, but on system memory instead. It offers up to 16 GB (compared to 64 GB on Curie). Please note that any data written to it is reduces the size of the memory that remains available for computations. In our case it change the number of core use for post-treatment like pack_output.
* The default time limit for a job submission is 2hours (7200s) contrary to 24h (86400s) on curie
* '''The SCRATCH space purge policy has been simplified: any file that stays 60 days without being read or modified will be purged''', as well as any directory that remains empty for 30 days; the purge policy does not depend on disk occupation threshold any longer
# Configurations #
'''Following configurations are technically ready (scientific validation in progress) to be used at Irene'''. On 2018/7/2, performances are slow.
* IPSLCM6.1.5-LR
* IPSLCM5A2.1
* LMDZOR_v6.1.5
* ORCHIDEE_trunk
* ORCHIDEE_2_0
* NEMO_v6_OMIP
'''Following configurations need some modifications'''
* LMDZORINCA_v6
* XIOS : After download : update arch XIOS for IRENE
{{{
cd modipsl/modeles/XIOS/arch
svn update
}}}
# How to use old configurations #
* you need to update files AA_make.gdef and w_h_i in util directory
{{{
cd modipsl/util
svn update AA_make.gdef
svn update w_h_i
}}}
--> if you receive an svn error with this specific command, extract a new modipsl and copy these two files from it
* you need to change your version of libIGCM
{{{
cd modipsl
mv libIGCM libIGCM_curie
svn co svn://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM
}}}
* you need to download arch files for IRENE for LMDZ, ORCHIDEE, INCA
{{{
cd modipsl
mkdir modele_arch
cd modele_arch
svn co svn://forge.ipsl.jussieu.fr/orchidee/trunk/ORCHIDEE/arch/ ORCHIDEE_arch
svn co http://forge.ipsl.jussieu.fr/inca/svn/trunk/INCA5/arch/ INCA_arch
svn co http://svn.lmd.jussieu.fr/LMDZ/LMDZ6/branches/IPSLCM6.0.15/arch/ LMDZ_arch
svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-2.5/arch/ XIOS_arch
mv ORCHIDEE_arch/*IRENE* ../modeles/ORCHIDEE/arch/.
mv INCA_arch/*IRENE* ../modeles/INCA/arch/.
mv LMDZ_arch/*IRENE* ../modeles/LMDZ/arch/.
mv XIOS_arch/*IRENE* ../modeles/XIOS/arch/.
}}}
* if you are working with INCA you need a file pre_proc_X64_IRENE.x
{{{
cd modipsl/modeles/INCA
cp pre_proc_X64_CURIE.x pre_proc_X64_IRENE.x
}}}
* create a new makefile
{{{
cd modipsl/config/***
mv Makefile Makefile_curie
../../util/ins_make
gmake clean
}}}
* now you can work as on Curie