open TELEMAC-MASCARET The mathematically superior suite of solvers

Hi everybody,

I would like to know what computer characteristics are required for TELEMAC use.

Let me explain:
I work each day on TELEMAC and I would like to invest in a good machine adapted to my needs. I do several multi-processor computing on large model (>500 000 nodes).

What advice would you give about RAM (Random Acces Memory), graphics card and hard disk (velocity) ?

Moreover, what quantity of processors do you recommand for multi-processsing mode? and it's best to have 1 processor with 2 cores or 2 processors with 1 cores?

Thanks in advance,

Have a good day,

Loann

Hi

You could find some info in the benchmark topic.
Nevertheless, some advice...

For the moment, there is no GPU version of Telemac so graphic cards are not "important" except for post-treatment of course...
processor/core is a hard subject because it's hard to find processors with one core today and the question is hardly link to the network...

A very interesting approach is the work of Uwe Merkel on a Telemac Linux Live Cluster which could give a good solution potentially scalable with desktop machines.
For a more intensive use, a cluster is a solution but then the budget is the most important point ...

hope this helps

Dear Loann,

About 4 months ago we had the same exact questions in our company and some of the key points, mostly personal and related to an empirical experience which maybe someone here can point as a misconception are:

• Its preferable Higher frequency cores over quantity of cores, even if #cores*frequency dummy-calc of raw power give the same results, this is due to mpi wall and clock times (huge in some cases) differences
• Prefer single-cpu over multi-cpu, for the previous reason.
• Please don't count on maximun Ghz (A.k.a turbo freq.) per core when choosing the CPU , this is a value for "peak" tasks rather than continuous heavy load simulations like Telemac or openFOAM, a conservative value would be, accounting the default P-state, 0.4*(min+max GHZ) per core.
• Linux installation, customization and overall performance (and tuning) is better.
• RAM for multicore heavy-load usage is highly important, please check the slots usage and manufacturers optimal config. for the machine. e.g. 32gb (distributed in 4*8gb slots for a xeon 10 cpu) made the trick for pre, post and processing in our case (models around >1-2M cells).
• In some cases virtual cores doesn't add any improvements (there is a wide discussion around this), we preferred to turn off the hyperthreading and almost nothing changed, other users reports a good scalability
• SSD makes a huge difference when e.g. post-processing and using Paraview and if the writing is a bottleneck in your simulation (have quite small graphic printout period), considering purchasing one
• A decent Quadro/Firepro is never a bad deal for pre and post processing, specially if you like to render and love the eyecandy-selling-videos for clients
• We are completely happy with the domain decomposition process of TELEMAC, but please be aware to have enough nodes per core, some recommends around ~50k nodes minimum per core to avoid the over-communication between cores, but there is no golden rule here.

As mentioned, I've been using TELEMAC for a short span of time so I cant really assure anything in this matter, so please take that into account.

regards,

José D.

Hi,

Thank you for your reply,

Considering finances of my business, I will try to identify the best method.

Have an excellent day,

Loann

Hello,

May I add my own experience:

• Telemac computations are usually memory bandwidth limited, so the higher the memory bandwidth, the better
• ARTEMIS with MUMPS solver, on the other side, is CPU power limited. Memory bandwidth does not make a difference
• A two socketed machine has twice the memory bandwidth than a single socketed. So it would be preferable to spend more on a two socketed machine and less on CPU GHz
• I may disagree with Jose's comment that "Its preferable Higher frequency cores over quantity of cores". In my opinion, as long as the CPU has enough power to saturate the memory channels, it doesn't make a big difference from how many cores that power comes. Of course there are cache latency penalties from using many cores but I don't think it justifies the added cost of frequency optimized CPUs. (Besides, frequency optimized CPUs are really targeting applications that license on a per core basis, where there is an incentive to get maximum power from as few cores)
• Have a look at this post. It gives an excellent overview of all aspects of the latest generation on XeonE5 CPUs
• Consider creating a mini-cluster with two nodes only using Infiniband Direct Connect without using an Infiniband switch that costs thousands €. I have read that you can even daisy-chain 3 nodes with two-port Infiniband cards at each node, but I haven't verified it with a professional

Best Regards,
Costas

Hi,

there is not "the right cluster". No matter what you buy, there is always another bottleneck. Currently (Q3/2016) the best price / performance ratio I found for Telemac is in the Xeon e5-26?? family. Don't go for more than 10 cores per CPU, as the RAM won't feed the additional data eaters.

The v3 and v4 generation will give you a speed up of up to 50% over v2, but for a price which is up to 5 times higher on the street market. If your not running it 365days / 24hours, using the older generation might result in much better economic efficiency.

I'm currently setting up several small clusters with 64 to 128 cores (Xeon E5) for 2 customers and my own office.
Above 8 nodes = 128 cores the electricity supply in my office is not strong enough ... 3600W max. I would need a new cable ...

My verdict for T2D and T3D (strongly varying with your models parameters!):

Small problems, lets say < 2000 mesh nodes per cpu core are strongly memory speed and memory channel width dependent.
But once you are on quad channel memory, the frequency is only important for the really small problems.

Large problems > 750 000 indeed are CPU frequency dependent. But this can be solved with just using: ... more CPUS ...

In between: Its is mostly the network latency that breaks your performance.

I tried 1G Ethernet vs. 10G Ethernet vs. QDR4 Infiniband.
First and second are only slightly different in latency. Last is 10 times better. Practical results: Medium size problems can be accelerated by up to 30% and more with QDR Infiniband, but the effect is best around 3 to 8 nodes with 16 cores.

Infiniband is the worst configuration hell ... Use Mellanox not Qlogic, as the first has much more resources in the www.

Partel and Gretel perform much better on SSDs with high IOPS rates than on raid systems. Paraview and other Postprocessors aswell. Eventhough the bandwidth is equal.
Think about the pcie-> m:2 ssd cards!

Parallel file systems like glusterfs are very easy to install.
This is great if you try to print out many time steps for videos.

If you are lucky, you can get a 4 or 8 node server (64 to 128 core Xeon E5) for 3500€ to 7000€. Used, leasing return hardware, with 1 year manufacturer warranty. Not a big risk.

Testing suitable hardware has a lot to do with trail and error. I ordered and returned almost anything twice .. Expect a lot of configuration problems. Or just ignore that you spend thousands of Euros on the wrong components.

If you need more infos, just contact me.

Kind regards,
Uwe

Hello,

I am also looking for advices for a personnal calculator.
My T3D configuration has around 100ke nodes. I need to perform long time simulations (one year or more, time step=25s)

I have a quotation for a 2*Xeon E5-2680v3 (2cpu*12=24cores) around 6ke. Is it a good choice ?

Thanks !
Vincent

Hello Vincent,

Why not a v4 Xeon? It is the latest version and has more memory bandwidth. Usually it is similarly priced to the v3.

Regards,
Costas