open TELEMAC-MASCARET The mathematically superior suite of solvers

Hi all,

I'm trying to export the time series file of a 3D hydrodynamic model. But the merging process fails. However, it works well with the 2D time series files. Could you please have a look at the error messages below?
Additional information:
- The requested 3D output parameters are Z,U, and V
- I'm running the model on a cluster, using 48 processors.
- In the The cas file, I have specified the following keyword/values:
2D TIME SERIES FILE ='2D_Hist.slf'
3D TIME SERIES FILE ='3D_Hist.slf'
2D TIME SERIES FILE FORMAT ='SERAFIN' / default
3D TIME SERIES FILE FORMAT ='SERAFIN' / default
TIME SERIES COORDINATES FILE='OutputPoints.txt'


Error message:

... merging separated result files

collecting: T3DHYD
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/gretel < gretel_T3DHYD.par >> gretel_T3DHYD.log
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
STOP 0
collecting: T3DRES
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/gretel < gretel_T3DRES.par >> gretel_T3DRES.log
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
STOP 0
collecting: T3DHI2
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/grehis_autop < grehis_T3DHI2.par >> grehis_T3DHI2.log
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
STOP 0
collecting: T3DHI3
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/grehis_autop < grehis_T3DHI3.par >> grehis_T3DHI3.log
double free or corruption (out)

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0 0x15555526ca12 in ???
#1 0x15555526bba5 in ???
#2 0x155554ccfdaf in ???
#3 0x155554d1c54c in ???
#4 0x155554ccfd05 in ???
#5 0x155554ca37f2 in ???
#6 0x155554ca412f in ???
#7 0x155554d26616 in ???
#8 0x155554d282ff in ???
#9 0x155554d2a954 in ???
#10 0x405c68 in ???
#11 0x4022c2 in ???
#12 0x155554cbaeaf in ???
#13 0x155554cbaf5f in ???
#14 0x4022f4 in ???
#15 0xffffffffffffffff in ???
Traceback (most recent call last):
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/telemac3d.py", line 7, in <module>
main('telemac3d')
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/runcode.py", line 288, in main
run_study(cas_file, code_name, options)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run_cas.py", line 169, in run_study
run_local_cas(my_study, options)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run_cas.py", line 75, in run_local_cas
my_study.merge(options.gretel_method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/study.py", line 704, in merge
self.ncsize, method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run.py", line 228, in run_recollection
globnd, ncsize, nplan, method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run.py", line 371, in run_grehis
+ '\n '+tail+'\n\n'+log)
utils.exceptions.TelemacException: Could not merge your file T3DHI3 (runcode=134) with the error as follows:

... The following command failed for the reason in the listing
/trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/grehis_autop < grehis_T3DHI3.par >> grehis_T3DHI3.log

Kind regards,
Mostafa

Hello Mostafa,

First thing is to read and/or upload the file grehis_T3DHI3.log which is the log file of the run of /trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/grehis_autop < grehis_T3DHI3.par

If you upload the file, please change the extension to e.g. .txt (grehis_T3DHI3.txt) as .log is not allowed on this forum.

Have you tried to run:
- the same computation in sequential if not too long?
- to run the same computation but with less graphic printouts?

If you cannot manage with this log file, as it seems to be a memory issue, you should compile another configuration with debug options, see e.g.
See e.g. the S11.gfortran.debug configuration in the $HOMETEL/configs/systel.edf.cfg configuration file, in particular the flag fflags_debug_gfo for gfortran compiler.
fflags_debug_gfo: -g -Wall -fcheck=all -fbacktrace -fbounds-check -finit-integer=-1 -finit-real=nan -ffpe-trap=invalid,zero,overflow

Then, in your temporary folder, run:
/trinity/shared/apps/telemac/v8p5r0/builds/parallel/bin/grehis_autop < grehis_T3DHI3.par

It will show where the issue may occur, the first suspicious line and may help you to change something in your computation.

How long is your computation? How many time steps? What is the graphic printouts period?
Can you upload the time series coordinates file OutputPoints.txt please?

Chi-Tuan

Hey Pham,

Thanks for the complete answer. I have gone through all of your recommendations one by one, and here are the results:

- Running in serial mode or reducing the graphic printout intervals didn't help. FYI, my model has 500K+ nodes, a time step of 3 sec, and I want to export the history points every 1 minute. The simulation period is 1 month.

- I found that the issue with the failed merging of the 3D history output files was due to one of the introduced output locations being outside the modeling domain. Although Telemac gives a warning at the first time step of the simulation for the outlier point, it fails in the merging step of the 3D files, but the 2D merging step was successful. By removing that specific point, the merging step was successful!

- We also checked a new compilation of Telemac using your suggested debugging option. The newly compiled version has a different behavior as it gives an error at the merging step of the 2D history points. See the message below, which is clear enough for the user to debug the model.

... merging separated result files

collecting: T3DHYD
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel-debug/bin/gretel < gretel_T3DHYD.par >> gretel_T3DHYD.log
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
STOP 0
collecting: T3DRES
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel-debug/bin/gretel < gretel_T3DRES.par >> gretel_T3DRES.log
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL
STOP 0
collecting: T3DHI2
+> /trinity/shared/apps/telemac/v8p5r0/builds/parallel-debug/bin/grehis_autop < grehis_T3DHI2.par >> grehis_T3DHI2.log
At line 408 of file /trinity/shared/apps/telemac/v8p5r0/sources/utils/gretel/grehis_autop.f
Fortran runtime error: Index '169' of dimension 1 of array 'x' above upper bound of 168

Error termination. Backtrace:
#0 0x15555526ca12 in ???
#1 0x15555526d509 in ???
#2 0x15555526da99 in ???
#3 0x4067f8 in ???
#4 0x4022c2 in ???
#5 0x155554cbaeaf in ???
#6 0x155554cbaf5f in ???
#7 0x4022f4 in ???
#8 0xffffffffffffffff in ???
Traceback (most recent call last):
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/telemac3d.py", line 7, in <module>
main('telemac3d')
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/runcode.py", line 288, in main
run_study(cas_file, code_name, options)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run_cas.py", line 169, in run_study
run_local_cas(my_study, options)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run_cas.py", line 75, in run_local_cas
my_study.merge(options.gretel_method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/study.py", line 704, in merge
self.ncsize, method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run.py", line 228, in run_recollection
globnd, ncsize, nplan, method)
File "/trinity/shared/apps/telemac/v8p5r0/scripts/python3/execution/run.py", line 371, in run_grehis
+ '\n '+tail+'\n\n'+log)
utils.exceptions.TelemacException: Could not merge your file T3DHI2 (runcode=2) with the error as follows:

... The following command failed for the reason in the listing
/trinity/shared/apps/telemac/v8p5r0/builds/parallel-debug/bin/grehis_autop < grehis_T3DHI2.par >> grehis_T3DHI2.log


Ideally, it would be more helpful if Telemac could skip the outlier point(s) and merge the history outputs without any error (might be fixed in the next release).

Thanks again for your help!

Mostafa