open TELEMAC-MASCARET The mathematically superior suite of solvers

Dear All,

I have a model running in parallel mode, I tried to run in on 2,4 and 8CPU,
The model run fine with 2 CPU. However when I assign 4 or 8 CPU the model crash (not at the same time step) with the following error message :

---

@STREAMLINE::SCARACT: THE NUMBER OF TRACEBACK INTERFACE CROSSINGS IGEN > 99

PLANTE: PROGRAM STOPPED AFTER AN ERROR
EXITING PVM: CALLING P_EXIT

EXITING MPI

RETURNING EXIT CODE: 2

---

I wonder if some of you already had the same error message and found a way to solve it? What will be the consequence if I modify the subroutine STREAMLINE.f by replacing the corresponding value 99 by 1000?

Thanks in advance for your help,

Mathieu

Hello,

You have this message when a characteristic path line crosses more than 99 sub-domain boundaries, which is very unlikely unless your case is crashing and velocity going to infinity. So the real reason is probably elsewhere, the first thing to look at being your specific fortran file if you have one.

JMH

Thanks a lot for your quick reply.

The model crashed after 6h30 to 10h of simulation. The results files doesn't show any infinite or very high value of velocity, despite the model is still 'warming up'.
I forgot to say that I am running a Telemac-3D model(12 layers), the boundaries are varying in time and space.

Including the above extra information, do you have any idea why the model work fine when I run it in 2CPU but crash with 4 or 8CPU?

Mathieu

Hello,

You could perhaps change the SCHEME FOR ADVECTION OF VELOCITIES to 4 (no tidal flats) or 14 (if tidal flats) to see if everything is correct without the method of characteristics.

JMH

Hello,

I changed the SCHEME FOR ADVECTION OF VELOCITIES to 14, the model stop crashing but the model take much (much) more time to run.
I notice that there is a high velocity at the boundary where the model is split due to the parallel mode(at least 4 time), there is also small Islands split in 2 (exactly where the high velocity occur), i think that could explain part of the instability, I'll try to modify my mesh to see what will happen....

Thanks a lot for your help,

Mathieu

Hello,

If you have abnormal high velocities on interfaces between sub-domains, this is a hint that something is wrong in parallelism, either in your specific programming (do you have special boundary conditions or specific programming ?) or in the code, in which case we would be interested in having the case.

With best regards,

Jean-Michel Hervouet

Hello,

Some sub-routine are modified, the main modification is made to implemented the boundary condition which is composed of free surface elevation varying in time and space (We used the same equivalent sub-routine for Telemac-2D model without any problem so far).
I'll try to add pictures in the following day, to illustrate the problem......

Kind Regards,

Mathieu

Hello,

I'd prefer seeing the modified subroutines to see if this could be the problem.

JMH

Hello,

I have a similar error when I'm running a Telemac-3D model. Additionally, I get several lines of the following message at most steps.

Finally, it comes to a certain timestep and the run exits with the following message:

I'm running the code on 32-bit ubuntu 11.10, 4-core CPU.

Also, this issue does not seem like an installation problem. So maybe the thread should be moved to Telemac-3D forum.

Avni

Hello,

In your case the number of iterations says that the problem is becoming hard somewhere, probably because of velocities growing. You can try to reduce the time-step or to locate where this happens (boundary condition, tidal flat,...)

JMH