open TELEMAC-MASCARET The mathematically superior suite of solvers

I am having some issues when using multiple water sources together with running Telemac 2d in parallel mode. The issues are as follows:
(1) The reported ADDITIONAL VOLUME DUE TO SOURCE TERMS is sometime quite a bit less than the sum of the water sources values
(2) The discrepancy is reported at the end of the run as “TOTAL VOLUME LOST”
(3) The actual total flow being inserted into the model in these cases may still actually be the correct total (as shown by the outflow when stabilized), or it may be less the amount inserted in the waters sources, e.g. water has disappeared.
(4) When the same model is run in non-parallel mode, there are no issues.

The problems tend to happen more with larger models, and with models that have more water sources. Depending the “ncsize” e.g. number of CPUs, the issues may or may not occur with a particular model, and errors in discharges will change depending on the number of CPUs used. e.g. a model may have problems with 16 CPUS but run fine with 8.

I created a sample model to illustrate. The model has 13 water sources, each water source corresponds to a separate node (13 nodes), and each is placed very close (within 1 millimeter) if it’s respective node. Each node has 10 M3/S of water being inserted, for a total of 130 M3/S. The water source nodes are not edge nodes, but they are all adjacent to edge nodes. This was the simplest model I was able to make that exhibited problems.
I set the time step at 10 seconds to be able to easily compute the reported additional volume due to source terms. It should report 130 M3/S x 10 seconds = 1,300 M3 at each iteration.
Here is what happens when I run this model on 16 CPUs e.g. –-ncsize=16:
(1) It reports 1,259.541 M3 in 10 seconds (e.g. 125.95 M3/S) due to source terms instead of 1,300 M3 (e.g. 130 M3/S).
(2) By the time run has stabilized at the end of the run, it is reporting about -128.20 M3/S of “FLUX BOUNDARY” the downstream end. The run is for 100,000 seconds, but running for 200,000 seconds barely budges this -128.20 number, e.g. the model is effectively steady-state by 100,000 seconds. So it appears that about 128.2 M3/S is actually being inserted in the model, compared the 130 M3/S intended, and 1.8 M3/S is not accounted for (130 – 128.2 = 1.8). Note the 128.2 M3/S value does match the printout value of 1295.5 M3 per time step (125.95 M3/S) being inserted either.
(3) At the end of the run of 100,000 seconds, “TOTAL VOLUME LOST” is reported as -234983 M3. This works out to 2.35 M3/S e.g 234,983 M3/S /100,000 seconds.

I’m curious if this a known issue regarding water sources and parallel runs, or if I’m doing something wrong. With other more complex models I’ve had even larger discrepancies, with 5% or more of the discharge lost. There is a remark in the documentation at the beginning of section 8 that “In parallel mode, the sources must coincide exactly with one point of the mesh, so this is recommended in all cases.” I’m not entirely sure what this statement means. I did place only one water source per node, and placed them very close to the node locations. The downstream end has fixed water elevations set in the boundary file, thus initially water backs into the model at the downstream end, then water arrives from the upstream end and the model eventually becomes quasi-steady state. Other details: Used the latest version 7p2r1, on Debian 9 Stretch. Gfortran and OpenMPI from the standard Debian repository were utilized in the build phase of the program. Manning roughness is specified in the selafin file (as the BOTTOM FRICTION array), which overrides the single value called out in the .cas file. Double precision “SERAFIND” format was used for both input and outputs.
I’ve included the model input files. The command I ran them with is:
telemac2d.py --ncsize=16 sources_model.cas

Here is the screen printout near the end of the run:

---

ADVECTION STEP

---

DIFFUSION-PROPAGATION STEP
GRACJG (BIEF) : 4 ITERATIONS, RELATIVE PRECISION: 0.8037713E-05
POSITIVE DEPTHS OBTAINED IN 10 ITERATIONS

---

BALANCE OF WATER VOLUME
VOLUME IN THE DOMAIN : 5524188. M3
FLUX BOUNDARY 1: -128.2040 M3/S ( >0 : ENTERING <0 : EXITING )
ADDITIONAL VOLUME DUE TO SOURCE TERMS: 1258.541 M3
RELATIVE ERROR IN VOLUME AT T = 0.1000E+06 S : -0.4257092E-05

---

FINAL BALANCE OF WATER VOLUME

RELATIVE ERROR CUMULATED ON VOLUME: -0.3220620E-01

INITIAL VOLUME : 0.000000 M3
FINAL VOLUME : 5524188. M3
VOLUME THAT ENTERED THE DOMAIN: -7296206. M3 ( IF <0 EXIT )
VOLUME ADDED BY SOURCE TERM : 0.1258541E+08 M3
TOTAL VOLUME LOST : -234983.1 M3

END OF TIME LOOP

EXITING MPI

CORRECT END OF RUN

ELAPSE TIME :
1 MINUTES
9 SECONDS

Hello

Sorry for this late reply.

We are aware of several issues of non-reproducibility between serial and paralllel runs. These issues are mainly due to the non-reproducibility of several elementary operations (such as associativity, matrix-vector product, etc.) We are working on this and have realized major advances these last couple of years.

In your case, the loss of mass of water introduced by sources is due to the default way with which Telemac handles sources. Actually, the default option for the keyword TYPE OF SOURCES is 1 which means a linear distribution on the elements around the source node (which is coherent with the P1 Finite element method used for the resolution of the full set of shallow water equations).
With this option, a non conservative assembling operation is acheved at the end of each iteration which causes these losses.
To overcome such issue, you can set TYPE OF SOURCES =2 which is the Dirac option. This option will concerve exactly the mass.

I hope that this helps

with my best regards

Riadh ATA

It works! Discharge is conserved. So far this is only issue I've run into using the parallel mode (which provides really impressive speed gains). And now I have a solution for this issue.

Thank you so much for your ongoing contributions and support of this excellent software.