Opened 3 months ago

Closed 3 months ago

#773 closed defect (fixed)

Mass balance error on PFT 9 in analytical spinup

Reported by: mmcgrath Owned by: luyssaert
Priority: major Milestone: ORCHIDEE 4.1
Component: Biogeochemical processes Version:
Keywords: Cc:

Description

When running with r7157 on Irene with a production executable and 127 CPUs, I have a mass balance error in year 4 of analytical spinup (FG1).

14pixel number to lat/lon:      6       69.0000000000       13.0000000000

14 Entering functional allocation growth
14 Error: mass balance is not closed in stomate_allocation for carbon
14  ipts, ivm;            6           9
14  Difference is,  -1.008621702631228E-005
14  pool_end,pool_start:    1.84495063822095        1.84495063822095
14
14FATAL ERROR FROM ROUTINE stomate_allocation
14 --> Mass balance error
14 -->
14 -->
14
14Fatal error from ORCHIDEE. STOP in ipslerr_p with code

I can continue running by making

ERR_ACT=1

in orchidee.def But should this be looked into more carefully?

Change History (5)

comment:1 Changed 3 months ago by mmcgrath

I have attempted to reproduce the problem on a single processor.

LIMIT_WEST=12
LIMIT_NORTH=70
LIMIT_SOUTH=68
LIMIT_EAST=14

while also setting ERR_ACT=3, so that the code crashes on mass balance errors. The code crashes in the same year and for the same PFT, though the magnitude of the difference is different.

 Entering functional allocation growth
 Error: mass balance is not closed in stomate_allocation for carbon
  ipts, ivm;            1           9
  Difference is,  -3.296556448420786E-005
  pool_end,pool_start:    4.74060613114703        4.74060613114703

FATAL ERROR FROM ROUTINE stomate_allocation
 --> Mass balance error
 -->
 -->

Fatal error from ORCHIDEE. STOP in ipslerr_p with code

I tried with a debug executable, and it crashes almost immediately in the first year.

[irene1187:80102:0] Caught signal 8 (Floating point exception)
==== backtrace ====
 2 0x000000000006bc9c mxm_handle_error()  /var/tmp/OFED_topdir/BUILD/mxm-3.7.3112/src/mxm/util/debug/debug.c:641
 3 0x000000000006c1ec mxm_error_signal_handler()  /var/tmp/OFED_topdir/BUILD/mxm-3.7.3112/src/mxm/util/debug/debug.c:616
 4 0x0000000000036450 killpg()  ??:0
 5 0x0000000001e87cf8 hydrol_mp_hydrol_main_()  /ccc/work/cont003/gen6328/mcgrathm/TRUNK.DEV/modeles/ORCHIDEE/build/ppsrc/sechiba/hydrol.f90:818
 6 0x000000000092a9ba sechiba_mp_sechiba_main_()  /ccc/work/cont003/gen6328/mcgrathm/TRUNK.DEV/modeles/ORCHIDEE/build/ppsrc/sechiba/sechiba.f90:1139
 7 0x00000000005b2730 intersurf_mp_intersurf_main_2d_()  /ccc/work/cont003/gen6328/mcgrathm/TRUNK.DEV/modeles/ORCHIDEE/build/ppsrc/sechiba/intersurf.f90:584
 8 0x0000000000511ef1 MAIN__()  /ccc/work/cont003/gen6328/mcgrathm/TRUNK.DEV/modeles/ORCHIDEE/build/ppsrc/orchidee_ol/dim2_driver.f90:1285
 9 0x000000000044d1ce main()  ??:0
10 0x0000000000022555 __libc_start_main()  ??:0
11 0x000000000044d0e9 _start()  ??:0
===================
/ccc/scratch/cont003/drf/mcgrathm/RUN_DIR/6914737_73363/FG1.MBD.r7157.73363/./run_file_RLffHX: line 6: 80102: Floating exception
srun: error: irene1187: task 0: Floating point exception

The lines of code it's referring to are:

       DO ji = 1, kjpindex
          ! Roots need to be available to have an effect on ks. Use the maximum
          ! root_depth as calculated earlier in hydrol_root_profile. The original
          ! code used zz which is the depth of the node. Hence the depth of the
          ! node where roots are still available is used here. 
 ----->         DO jsl = 1, root_depth(ji,jv,inode)
             IF(soiltile(ji,jst) .GT. min_sechiba) THEN
                kfact_root(ji,jsl,jst) = kfact_root(ji,jsl,jst) * &
                     & MAX((MAXVAL(ks_usda)/ks(njsc(ji)))**(- vegetmax_soil(ji,jv,jst)/2 * &
                     (humcste(jv)*zz(jsl)/mille - un)/deux), un)
             ENDIF
          ENDDO
       ENDDO

comment:2 Changed 3 months ago by mmcgrath

Attempted to fix it in r7160 by moving the calculation of kfact_root to just after the calculation of root_depth, but before kfact_root is used in hydrol_soil. Two years have run with no crash, as opposed to a crash immediately in the first year.

comment:3 Changed 3 months ago by mmcgrath

Using r7160 on Irene in debug mode for the above test case on a single pixel results in a similar error in year 4, although once again the magnitude of the difference is a not exactly the same.

0 Entering functional allocation growth
0 Error: mass balance is not closed in stomate_allocation for carbon
0  ipts, ivm;            1           9
0  Difference is,  -4.670998382867592E-005
0  pool_end,pool_start:    4.73556467966789        4.73556467966789
0
0FATAL ERROR FROM ROUTINE stomate_allocation
0 --> Mass balance error
0 -->
0 -->
0
0Fatal error from ORCHIDEE. STOP in ipslerr_p with code

comment:4 Changed 3 months ago by luyssaert

  • Owner changed from somebody to luyssaert
  • Status changed from new to assigned

The problem was related to how dark respiration was accounted for when plant_status was isenescent or ipresenescence. That part of the code was only correct for plant_status = icanopy. The problem has been fixed in r7167.

comment:5 Changed 3 months ago by luyssaert

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.