open TELEMAC-MASCARET The mathematically superior suite of solvers

Hi,

we are running a research-stage 3d model of a reservoir to assess effects of climate change on reservoir sedimentation. Thus, we want to model quite long time ranges for different climate change scenarios and our models take ages for running, even on latest computers. To downsize calculation runtime, we are currently working with openmpi+metis. The installation section also mentions mpich, hpc schemes, or mumps, which might potentially speed-up calculations.

Does anyone have recommendations for libraries and compiling options that are computationally efficient for Telemac (also if those refer to 2d and/or Gaia)?

In addition, what solver can be recommended with regards to calculation speed?

Thanks,
Sebastian

Hello Sebastian,

MUMPS is currently useful with ARTEMIS in parallel mode, when matrices are not sparse. With TELEMAC-3D, it has not been optimised yet.

Which compiler do you use? Intel may be more efficient than gfortran (it is the case for the clusters I use).

Maybe you should install TELEMAC with the less external libraries that you really need. E.g. AED2, GOTM, MED, MUMPS are not always necessary.

Using optimisation option -O2 is OK (-O3 may be hazardous).

You can have a look at the systel.edf.cfg configuration file with some examples for clusters (e.g. gaia.intel.dyn or cronos.intel.dyn).


Depending on what kind of matrices you want to invert, you can change the solver. You can read some recommendations in the appendix E of the T3D user manual and what can be chosen as solver. For the diffusion step, if using conjugate gradient solver (efficient with symmetric matrices), you can use preconditioning 34 rather than 2 to speed-up the solving).
If you use GMRES, you can play with the dimension of the Krylov spaces (OPTION OF SOLVER FOR...): the bigger the more accurate, but also the longer.

Do you have tidal flats?

Which turbulence model do you use? If using k-epsilon or another transport equation turbulence model, you can try to downgrade your computation with a mixing length model on the vertical and constant viscosity for the x and y directions if the results are OK.

You may have to optimize your mesh (2D and the number and distribution of planes).

You can also take 1. for the different MASS-LUMPING parameters and IMPLICITATION = 0.9 e.g. (not 1.).

Which advection scheme(s) do you use?

Maybe you can provide your current steering file and users may suggest you some ideas?

You can have a look at the WAQTEL example called heat_exchange which gives you an example of a reservoir modelling.

Hope this helps,

Chi-Tuan

Hi Chi-Tuan,

Thanks, this is great advice.

Yes, we are having tidal flats ON, and we are using k-epsilon suggesting that it is a generalization of the mixing length model.

For implicitation we use 0.55 (this is default since v8 isn't it?), but no MASS LUMPING. I am not sure if you mean that decreasing implicitation decreases computation time or vice versa?

For advection we generally use scheme 5 with option 4 (not 2 because of tidal flats).

The optimization option looks interesting, I will take a deeper look at it.

I attached our baseline version of the 3d steering file:

- and here is also the 2d baseline file:


We use both files as templates for modeling hydrodynamics of gravel-cobble bed rivers, optionally with a reservoir. This is why the links to input geometry files have quite random names.

In addition, I am testing to mesh with BASEMENT's baseMESH plugin in QGIS (as follows hydro-informatics.com/numerics/pre-qgis-bm.html ), because of better cross-platform compatibility of QGIS (compared with BK). Though we did not yet test what is the difference in computation time between both meshing algorithms. I also wanted to work with Salome and the Medfile library, but Salome-Hydro on Debian 10 is buggy.

Best,
Sebastian

Hello Sebastian,

Some default values were changed a few years ago because with Agnes (former colleague of mine), we thought that when using TELEMAC-3D, you have a correct computer or cluster to run it and saving CPU time is not the first criterion, but getting more accuracy is. That is why some default values of keywords were changed (in particular less implicit but semi-implicit which should be more accurate, use of the LIPS schemes (SCHEME FOR ADVECTION OF = 5 + SCHEME OPTION FOR ADVECTION OF = 4) which are less diffusive and work with tidal flats, etc.
Using MASS-LUMPING... = 1. may solve CPU time as some matrices are then diagonal... but the results may be smoother.

Anyway, you can change them of course if you want to decrease CPU time but you should check the results are OK enough (balance between accuracy and CPU time).

What do you try to model? If there is no tracer and only interested in pure hydrodynamics, you can try the method of characteristics for velocities and k-epsilon to save time for example. If there are tracers, you have to keep a conservative scheme that works with tidal flats (LIPS is less diffusive, but if using NERD schemes - 13 or 14 - is OK, you can try it, but it is more diffusive), but you can also try to mix with characteristics for velocities and k-epsilon.

If the tidal flats area are not too big, you can change the local elevation where there are tidal flats to always have water and use other advection schemes which work when there is always water.

Using a mixing length model over the vertical reduces the number of transport equations to solve, to save CPU time but you have to calibrate it and this may be less accurate.

When I wrote 2D optimisation of the mesh, I wanted to say decrease the number of 2D elements, not necessary to change your mesh generator.

That is why I can suggest you to downgrade your computation and save CPU time. Maybe other users can suggest other ideas?

Hope this helps,

Chi-Tuan

Thanks Chi-Tuan, that is valuable input again. Though your already provide many possible responses to the answer to your question: We are having suspended load (i.e. tracers enabled). Accuracy is an issue and the last aspect we want to downgrade, so this is why I very much appreciate your first suggestions on compiling TELEMAC.

Any other feedback is very welcome.

Best,
Sebastian

Hello again,

I agree with you if accuracy is an issue.

In that case, for the steering file, what I have already suggested and may not change accuracy (or not too much) are:
- use of the preconditionning = 34 when possible (i.e. with conjugate gradient, SOLVER FOR... = 1) which should accelerate the solving (you can also change the SOLVER for PPE and propagation to conjugate gradient, and preconditionning for PPE = 34 in that case if it accelerate your computation,
- use MASS-LUMPING = 1. for DEPTH and DIFFUSION (some are mandatory with some combo of options),
- you can try different IMPLICITATION up to 0.9 (not too big) and see your results and CPU times,
- if you spend a lot of iterations in some steps of the algorithm (e.g. PPE, propagation, diffusion of velocities), you can sligthly relax the ACCURACY FOR..., i.e. to 1.E-7 or 1.E-6 if it does not change your results too much. For the tracers, I would still recommend to keep at the default value 1.E-8 or less.

Changing the advection schemes, turbulence model, wet area will have an impact on your results for sure.

Hope this helps,

Chi-Tuan

Thanks again. Out of curiosity do you, or does anyone have an opinion on Telemac's finite volume scheme?

Best,
Sebastian

Hello,

Just a quick comment. Chi-Tuan advices for accurary and performance are very good. Regarding pure performance, I think that the installation and compilation can have a larger effect on CPU times than different numerical settings, so spending time finding the optimal install/compilation set-up is important in your case I think, especially if working on a HPC system. I am however not expert in this field and can't give any trusty tips...

As for finite volume scheme, it is only available in 2D.

Kind regards
PL

Hello,

As Pierre-Louis wrote, Finite Volumes schemes are only available for TELEMAC-2D, not TELEMAC-3D.
My colleagues Fabien Souillé and Florent Taccone did a great job to rebuild the Finite Volumes module in TELEMAC-2D for the latest release (v8p2) and corrected several bugs. You can use them from this release (but please do not for previous releases).

They are particularly useful when having shocks or wet/dry issues. You can have a look at the TELEMAC-2D validation manual, e.g. dambreak examples (Ritter and Stoker, malpasset), Thacker examples or flume_tracer examples.

Hope this helps,

Chi-Tuan