open TELEMAC-MASCARET The mathematically superior suite of solvers

Hello,

Is there a study on Telemac scaling with very large meshes, e.g. about 10 million elements. I know there is a 2011 paper on 2m-element mesh for a variety of platforms (www.sciencedirect.com/science/article/pii/S0045793011002131), but I was wondering if anyone has documented performance on larger problems and on more cores.

Kind regards,

Alexandros

Hello,

we have smaller meshes between 0.5 and 2.5 million elements. The performance depends on the equations and modules you use.
The specific cluster and network is also important. But be careful, memory speed is maybe more important than the clockrate/frequency of the cpu's. Especially newer CPUs allow fine grained dynamic adaptions to the frequency (turbo-mode). This makes bench marking a horrible task.

Telemac is often regarded as one of the fastest codes. Even we observed that scaling is in some cases not perfect (we observed some MPI-waits) I guess there are not many alternatives. We observed that the speed depends on cluster, network, memory,.... and the Telemac Version. There are many tools to analyze the performance on a given cluster and a specific application. But it is a lot of work to study this since you need different clusters and money.

In my opinion it is easier to look at the ratio (simulated time)/(execution time). For larger meshes this ratio will usually decrease, in our cases the ratio is often about ~120 for large models. This means that you can simulate 120 days in one day. The efficiency limit is often somewhere near 1000 Elements per MPI-process. This means that you will need a cluster with ~1000 cores for your tests.

Best regards,

Leo

Just one point you have to keep in mind.
For the moment, the main limitation for very large mesh is the integer size of some values like the number of points and the number of elements...
Regards

Hello Alex

I have several big models with 4 million elements (finite volumes) and 12.6 million (coastal, finite elements).
As discussed above, the performance depends on the equations you solve and the numerical features involved in the resolution (model with wet/dry will not behave as a model without tidal flats, a model with strongly non linear source terms (Boussinesq for instance) will not behave as a classical SWE model.

Statistics given by Leo are mostly for coupled hydro-sediment models. Based on my experience (which is far from being exhaustive) a ratio of 5000 elements per MPI process is more optimal. With fewer elements shows a a start of slope break in the speedup curve.

I hope that this helps.

with my best regards
Riadh

Thank you, everyone, for your help and useful information.

riadh, might I enquire what modules you used for your larger coastal, finite-element model, and on how many cores did you run it?

Given that Telemac is one of the most popular coastal simulation codes, dare I propose a user-community database where members can upload run information (platform, number of cores, number of elements, number of vertices, version, wall time, simulated time, Telemac modules used, brief descriptor and date of run) in order to form a landscape of Telemac use and scaling? The landscape could be visualised in an attractive online plot, where each entry could is hyperlinked to papers etc.

Kind regards,

Alexandros

Hi Alexandros,

there is a benchmark section in the forum. I think the idea is great, but it means a lot of work for the users. Maybe it is also interesting to ask the users before the TUC about their models?


O.T.
From my point of view it is more interesting to have 3-4 different large benchmarks to compare different clusters.


best regards,

Leo

Hello Alex

I used Telemac-2D with shallow water equations and finite elements kernel.
For this model I did not achive a speed up study since I ran the model for 6h (physical time) on 24 cpu and it took roughly 16 min.

The idea of a model database is being discussed these last weeks. We are starting the database internally within the consortium members and our colleague Yoann is in charge of it. Maybe he could shed the light on it. The extension of such database to the whole community could be discussed in the next scientific community or during user club.

kind regards

Riadh