open TELEMAC-MASCARET The mathematically superior suite of solvers

Did you check the listing for all processor ?

I see that your are limiting the ram maybe this could be the issue ? Could you try increasing it.

By the way do you know that you can automatise the generation of the PBS file (You can have a look at eole.intel in systel.edf.cfg)

@yugi: Normally, when a process runs out of memory, we get a relevant error message in stderr. This is not the case, but worth examining.

@Sardar: You have two options to increase the memory per process, but I am unaware of how much RAM you really anticipate for your specific experiment:

1. using thin nodes (188 gb per node), you would lower ppn and increase pmem, so that pmem x pmem <= 188gb. I leave it to you to experiment
2. alternatively, you can use the bigmem nodes (760 gb per node), and keep ppn=36 and raise pmem to 21gb. If that is still not enough, you can switch to using superdome machine which offers up to 50gb per core/process

Have you ever run similar tasks on other machines/clusters successfully to the end? If so, how much memory did you typically need?

We usually take the whole node so we do not have to specify the memory.

Nevertheless, the amount of available memory is huge.
So maybe this is an indicator of a memory leak...

Dear Yugi and Coulet,

Thank you very much for your support! I will check simulations by allocating more memory. Let see what happens.

@Yugi I also checked the listing for all processors, there is no error message except some processors are listing ahead than others.

Dear all,

I am sorry to reopen this topic since I am encountering this problem again and cannot solve it yet. We use "slurm" in our university's HPC. Ihis problem occurs not in all simulation but only in very few simulations. We updated the slurm version and the new slurm system needs memory allocation. Previously, we used to take the whole node's resource. Do you find any solution or work around for this? Thank you very much in advance.

Best regards,
Htun.