open TELEMAC-MASCARET The mathematically superior suite of solvers

I want to simulate incoming tides in a large Gulf using the TPXO option. In the past, I have been doing so succesfully using Telemac 2D 8p0 on a HPC cluster. However, when I switch versions to 8p2, the simulation won't start correctly. Right before starting the first iteration, Telemac seems to stall but without any error message. When I want to run the same simulation but with a liquid boundary file instead of a TPXO boundary, it does work.

I added the output file as an attachment,after using the debugging = 1 keyword in the steering file.

Please let me know if I should add additional info.

Hello Ignace,

If you do not see any error message but your computation stops, try:
- 1st to run it with 1 processor (scalar mode),
- 2nd look at the PE*LOG files in the temporary directory which are listing file for every subdomain. Some of them may have different sizes compared to the others, the ones with bigger size are often suspicious with more writings (in particular error messages that may not be written for every subdomain).

If nothing special, please upload your steering file at least.

Hope this helps,

Chi-Tuan

Dear Chi-Tuan

Thank you for you reply.
I tried both options:

1: On one processor it runs perfectly on the cluster and so the issue is related to the parallel use.
2: All the PE*LOG files have the same size and all end with the line '- ACQUIRING LEVELS when initiating TPXO.

I added the steering file and such PE*LOG file.

Kind regards,
Ignace

Hello Ignace,

Can you change the .log extension to .txt extension so that we can read the PE00027-00027 file please? As .log extension is not allowed by this forum.

Anyway, looking at your steering file does not raise big issues. Can you upload the whole input files to run your computation?

Just 2 remarks about your steering file:
- be careful with OPTION FOR THE DIFFUSION OF VELOCITIES = 2 when dealing with tidal flats,
- it seems strange to me to use COEFFICIENT TO CALIBRATE TIDAL VELOCITIES = 0, that mean you want to prescribe tidal velocities = 0 at the maritime boundaries?

Chi-Tuan

Thank you for the tips. I added the PE file as txt.

Indeed, we are ignoring the prescribed tidal velocities at the moment, mainly for stability reasons.

Hello Ignace,

Have you tried with a different number of subdomains for your parallel computation? E.g: 2 or different values between 2 and 28? Do you have the same issue?
It seems that your computation stops between the reading of the 1st binary tidal file and the interpolation of data, there is no obvious reason why it stops (or maybe it takes long time to do and your computation is still running?).

I do not know what you prescribe in your boundary conditions file, but if not prescribing tidal velocities, the codes 544 should be used (and nothing special for the keyword COEFFICIENT TO CALIBRATE TIDAL VELOCITIES as free velocities at open boundaries).

Anyway, without seeing much about your input files, hard to help you, it would be better to share all your input files to run your model.

Chi-Tuan

Hi Chi-Tuan

I corrected the cli file, thanks for the tip. It didn't make a difference.
Also trying with a different number of subdomains did not work. In the meantime, we also tried to run it on another cluster with the same problem appearing.

here are the files:
drive.google.com/file/d/1-o_2kaYYk53GfPW...uA0/view?usp=sharing

Kind regards,
Ignace

Hello Ignace,

I succeeded in running your computation:
- on the most recent cluster I have access, without any problem and quickly (around 3 min),
- on another cluster but with more CPU time due to the reading or the interpolation of the tidal solutions to your tidal boundary nodes (around 5 h),
- on my laptop without any problem and quickly (around 2 min).

Anyway, for only the 3rd cluster I tried, the computation did not end after 8 h was still blocked during the reading or interpolation of the tidal solution.
I tried with lighter tidal solutions like TPXO 7.2 and it worked without any problem.

I think there is an issue of memory for the combo cluster you use and size of the global tidal solution.
Temporarily, I would suggest you to use TPXO 7.2 on your cluster or try to find another cluster.

Hope this helps,

Chi-Tuan

Hi Chi-Tuan

Thank you for testing.

I requested TPXO7 but I got the reply that they do not longer distribute this version. Is there any chance you could share it with me?

Nevertheless, we would really like to use the latest version on the long term. Do you have any idea how this can be related to the new version of Telemac?

Kind regards,
Ignace

Dear Chi-Tuan,

Thank you for your help so far. I am a colleague of Ignace working on the same project.

We don't have access to TPXO 7.2, which anyway doesn't have the resolution we need for our project.

We have tested our setup on different clusters we have access to (at different universities, on very different infrastructures and with the help of different IT teams), the same issue occurred.

We haven't had this problem with previous versions of Telemac (latest we used is 8.0), but we really need a new feature of the latest release 8.2, that is, the minimum depth to compute tidal velocities boundary conditions.

We would really appreciate a longer term solution to run Telemac 8.2 with TPXO 8 or 9.

Thank you again for your support.