276 | | ## Start or restart post processing jobs ## |
277 | | Please look at the next paragraph. |
278 | | |
279 | | # Start or restart post processing jobs # |
280 | | |
281 | | You can run post processing jobs once the main job is finished (for example if the post processing job was deactivated in [#Lespost-traitementsdansconfig.card config.card] or if you encountered a [#Debug bug]). |
282 | | |
283 | | On TGCC, the machine used for post processing is the same as the computing machine. On IDRIS, the machine to be used for post-processing is adapp (since July 2013) with same file system available : $WORKDIR, $HOME, ... . You can : |
284 | | 1. work directly in the experiment directory (which looks like `PATH_MODIPSL/config/IPSLCM5A/ST11/`). Possible option but not recommended. |
285 | | 1. work in a dedicated directory located in the experiment directory (e.g. `PATH_MODIPSL/config/IPSLCM5A/ST11/POST_REDO`). Best choice. |
286 | | 1. work in a dedicated directory which is independent of the experiment directory (e.g. `$WORKDIR/POST_REDO`). |
287 | | |
288 | | For the last two options you must first: |
289 | | * If your post processing directory is a subdirectory located in the experiment directory, copy (or make a link `ln -s`) the files and directories `config.card` `POST`, `COMP` and run.card if you want to post process only part of the simulation |
290 | | * If your post processing directory is an independant directory : |
291 | | * create a dedicated directory (for all simulations) ; |
292 | | * transfer libIGCM and run ins_job (for all simulations) ; |
293 | | * create a directory for the simulation to analyze (i.e. for each simulation) : |
294 | | {{{ |
295 | | #!sh |
296 | | cd $PATH_MODIPSL/config/IPSLCM5A/ST11 |
297 | | mkdir -p POST_REDO |
298 | | cd POST_REDO/ |
299 | | cp -pr ../COMP . |
300 | | cp -pr ../POST . |
301 | | cp -pr ../config.card . |
302 | | cp -pr ../run.card . |
303 | | }}} |
304 | | |
305 | | |
306 | | [[NoteBox(warn, Before submitting a post processing job at TGCC (`rebuild_fromWorkdir.job`\, `pack_debug.job`\, `pack_output.job`\, `pack_restart.job`\, `monitoring.job`\, `create_ts.job`\, `create_se.job`) you must make sure that the submission group in present in the job header (#MSUB -A genxxxx). If it isn't\, add it., 600px)]] |
307 | | ## Restart REBUILD ## |
308 | | |
309 | | * Copy the `rebuild_fromWorkdir.job` file to the experiment directory or to the dedicated directory; |
310 | | |
311 | | * Edit it: |
312 | | {{{ |
313 | | #!sh |
314 | | StandAlone=true |
315 | | |
316 | | libIGCM= # Points to the libIGCM directory of the experiment |
317 | | |
318 | | PeriodDateBegin= # beginning date of the last serie to be "rebuilded" |
319 | | |
320 | | NbRebuildDir= # Number of directories in the series to be "rebuilded" |
321 | | # until the PeriodDateBegin |
322 | | |
323 | | |
324 | | REBUILD_DIR= # Path for the backup of files waiting to be reconstructed |
325 | | # (looking like $SCRATCHDIR/IGCM_OUT/.../JobName/REBUILD or $SCRATCHDIR/TagName/JobName/REBUILD for version older than libIGCM_v2.0 |
326 | | # if RebuildFromArchive=NONE) |
327 | | |
328 | | MASTER=${MASTER:=curie|ada} # Select the computing machine : MASTER=curie for example |
329 | | }}} |
330 | | |
331 | | |
332 | | * Submit the job: |
333 | | {{{ |
334 | | #!sh |
335 | | ccc_msub rebuild_fromWorkdir.job # TGCC |
336 | | |
337 | | llsubmit rebuild_fromWorkdir.job # IDRIS |
338 | | }}} |
339 | | |
340 | | |
341 | | [[NoteBox(note, The rebuild job submits `pack_output.job` automatically., 600px)]] |
342 | | ## Restart Pack_output ## |
343 | | |
344 | | The pack_output (e.g. in case it was not submitted by the rebuild job): |
345 | | |
346 | | * Copy the `libIGCM/pack_output.job` file to the experiment directory or to the dedicated directory; |
347 | | * Edit it : |
348 | | {{{ |
349 | | #!sh |
350 | | libIGCM=${libIGCM:=::modipsl::/libIGCM} # path of the libIGCM library |
351 | | |
352 | | MASTER=${MASTER:=curie|ada} # machine on which you work |
353 | | |
354 | | DateBegin=${DateBegin:=20000101} # start date of the period to be packed |
355 | | |
356 | | DateEnd=${DateEnd:=20691231} # end date of the period to be packed |
357 | | |
358 | | PeriodPack=${PeriodPack:=10Y} # pack frequency |
359 | | }}} |
360 | | * Submit the job: |
361 | | {{{ |
362 | | #!sh |
363 | | ccc_msub pack_output.job # TGCC |
364 | | |
365 | | llsubmit pack_output.job # IDRIS |
366 | | }}} |
367 | | |
368 | | [[NoteBox(note, `create_ts.job` and `create_se.job` are submitted automatically., 600px)]] |
369 | | |
370 | | ## Restart Pack_restart or Pack_debug ## |
371 | | |
372 | | * Copy the libIGCM/pack_debug.job and libIGCM/pack_restart.job files to the experiment directory or to the dedicated directory; |
373 | | * Edit them : |
374 | | {{{ |
375 | | #!sh |
376 | | libIGCM=${libIGCM:=::modipsl::/libIGCM} # path of the libIGCM library |
377 | | |
378 | | MASTER=${MASTER:=curie|ada} # machine on which you work |
379 | | |
380 | | DateBegin=${DateBegin:=20000101} # start date of the period to be packed |
381 | | |
382 | | DateEnd=${DateEnd:=20691231} # end date of the period to be packed |
383 | | |
384 | | PeriodPack=${PeriodPack:=10Y} # pack frequency |
385 | | }}} |
386 | | * Submit the two jobs: |
387 | | {{{ |
388 | | #!sh |
389 | | ccc_msub pack_debug.job ; ccc_msub pack_restart.job # TGCC |
390 | | |
391 | | llsubmit pack_debug.job ; llsubmit pack_restart.job # IDRIS |
392 | | }}} |
393 | | |
394 | | |
395 | | |
396 | | ## Restart the Time series ## |
397 | | |
398 | | [[NoteBox(tip, In case you haven't done it yet\, retrieve `config.card` `COMP` `POST` and eventually `run.card` (post process only part of the simulation) in the `POST_REDO/` directory., 600px)]] |
399 | | |
400 | | There are two ways: |
401 | | |
402 | | ### !TimeSeries_checker.job - Recommended method ### |
403 | | |
404 | | * Copy the `libIGCM/TimeSeries_Checker.job` file to the experiment directory or to the dedicated directory; |
405 | | * Edit it: |
406 | | {{{ |
407 | | #!sh |
408 | | libIGCM=${libIGCM:=...MYEXP/modipsl/libIGCM} # Path of the libIGCM library |
409 | | |
410 | | SpaceName=${SpaceName:=DEVT} |
411 | | |
412 | | ExperimentName=${ExperimentName:=pdControl} |
413 | | |
414 | | JobName=${JobName:=MYEXP} |
415 | | |
416 | | CARD_DIR=${CARD_DIR:=${CURRENT_DIR}} # Path of the experiment directory |
417 | | # (including CURRENT_DIR if you copied |
418 | | # TimeSeries_Checker.job properly) |
419 | | |
420 | | export BRIDGE_MSUB_PROJECT=gen2211 # number of your genci project |
421 | | |
422 | | }}} |
423 | | * Run the !TimeSeries_Checker.job in interactive mode. It will call the missing create_ts jobs : |
424 | | {{{ |
425 | | #!sh |
426 | | ./TimeSeries_Checker.job |
427 | | }}} |
428 | | or alternatively, in ksh : |
429 | | {{{ |
430 | | #!sh |
431 | | ./TimeSeries_Checker.job 2>&1 | tee TSC_OUT # Create log file |
432 | | |
433 | | grep Batch TSC_OUT # find all the submitted jobs |
434 | | }}} |
435 | | |
436 | | |
437 | | ### Restart create_ts.job ### |
438 | | |
439 | | * Copy the `libIGCM/create_ts.job` file to the experiment directory or to the dedicated directory; |
440 | | * Edit it: |
441 | | {{{ |
442 | | #!sh |
443 | | StandAlone=true |
444 | | |
445 | | libIGCM= # Path of the libIGCM library |
446 | | |
447 | | PeriodDateEnd # end date of the time series to be created |
448 | | |
449 | | CompletedFlag # end date of the existing time series |
450 | | |
451 | | TsTask=2D # select 2D or 3D |
452 | | |
453 | | RebuildFrequency=true |
454 | | }}} |
455 | | * Run the job: |
456 | | {{{ |
457 | | #!sh |
458 | | ccc_msub create_ts.job # TGCC |
459 | | |
460 | | llsubmit create_ts.job # IDRIS |
461 | | }}} |
462 | | |
463 | | [[NoteBox(note, If your time series (TS) are 2D and 3D you must run the create_ts jobs twice and change the !TsTask variable accordingly., 600px)]] |
464 | | |
465 | | ## Restarting the seasonal mean calculations ## |
466 | | |
467 | | [[NoteBox(tip, Transfer `config.card`\, `COMP`\, `POST`\, and `run.card` (post process part of the simulation only) in the `POST_REDO/` directory if you have not done so yet., 600px)]] |
468 | | |
469 | | There are two methods: |
470 | | |
471 | | ### SE_Checker.job (recommended method) ### |
472 | | |
473 | | * Copy the `libIGCM/SE_Checker.job` file to the experiment directory or to the dedicated directory; |
474 | | * Edit it ; |
475 | | * Run `SE_checker.job` in interactive mode. This will call the create_se jobs: |
476 | | {{{ |
477 | | #!sh |
478 | | ./SE_Checker.job |
479 | | }}} |
480 | | or alternatively, in ksh : |
481 | | {{{ |
482 | | #!sh |
483 | | # Create logfile: |
484 | | ./SE_Checker.job 2>&1 | tee SE_OUT |
485 | | # Find all started jobs : |
486 | | grep Batch SE_OUT |
487 | | }}} |
488 | | |
489 | | ### Restart create_se.job ### |
490 | | |
491 | | * Copy the `libIGCM/create_se.job` file to the experiment directory or to the dedicated directory; |
492 | | * Edit it: |
493 | | {{{ |
494 | | #!sh |
495 | | StandAlone=true |
496 | | |
497 | | libIGCM= # path of the libIGCM library |
498 | | |
499 | | PeriodDateEnd= # end date of the decade to be processed |
500 | | }}} |
501 | | * Submit the job: |
502 | | {{{ |
503 | | #!sh |
504 | | ccc_msub create_se.job # TGCC |
505 | | |
506 | | llsubmit create_se.job # IDRIS |
507 | | }}} |
508 | | |