Welcome, Guest
Username: Password: Remember me

TOPIC: Unable to run particle tracking in parallel

Unable to run particle tracking in parallel 1 year 2 months ago #43095

  • N_Strahl
  • N_Strahl's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 12
I am running a particle simulation with 26415 particles. One for each raster centroid of my model DEM raster. To initialize my particles I use the USER_FLOT.f subroutine. I first check if the point in question is inside of the model boundary and then add it with ADD_PARTICLE. This works great on one processor. The problem is that when running in parallel I am getting the error: 'PARTICLE IN 4 SUB-DOMAINS', when running on 4 processors, for example. I am guessing this is because some of my particles are located on the partition boundaries, right? Is there a way to maybe add a partible to one partition only to ensure the error is not thrown? I'm also open to other suggestions.
Below is my fortran file and case file. Thanks in advance!
!                   ********************
                    SUBROUTINE USER_FLOT
!                   ********************
!
     &(XFLOT,YFLOT,NFLOT,NFLOT_MAX,X,Y,IKLE,NELEM,NELMAX,NPOIN,
     & TAGFLO,CLSFLO,SHPFLO,ELTFLO,MESH,LT,NIT,AT)

      USE BIEF
      USE STREAMLINE, ONLY : ADD_PARTICLE,DEL_PARTICLE
      USE ALGAE_TRANSP
!
      USE DECLARATIONS_SPECIAL
      IMPLICIT NONE

      INTEGER, INTENT(IN)             :: NPOIN,NIT,NFLOT_MAX,LT
      INTEGER, INTENT(IN)             :: NELEM,NELMAX
      INTEGER, INTENT(IN)             :: IKLE(NELMAX,3)
      INTEGER, INTENT(INOUT)          :: NFLOT
      INTEGER, INTENT(INOUT)          :: TAGFLO(NFLOT_MAX)
      INTEGER, INTENT(INOUT)          :: CLSFLO(NFLOT_MAX)
      INTEGER, INTENT(INOUT)          :: ELTFLO(NFLOT_MAX)
      DOUBLE PRECISION, INTENT(IN)    :: X(NPOIN),Y(NPOIN),AT
      DOUBLE PRECISION, INTENT(INOUT) :: XFLOT(NFLOT_MAX)
      DOUBLE PRECISION, INTENT(INOUT) :: YFLOT(NFLOT_MAX)
      DOUBLE PRECISION, INTENT(INOUT) :: SHPFLO(3,NFLOT_MAX)
      TYPE(BIEF_MESH) , INTENT(INOUT) :: MESH

      INTEGER ROW
      INTEGER COL
      INTEGER, PARAMETER :: ROWS = 564
      INTEGER, PARAMETER :: COLS = 869
      INTEGER N1,N2,N3,IELEM
      DOUBLE PRECISION, PARAMETER :: EPSILO = 1.D-10
      DOUBLE PRECISION DET1,DET2,DET3,ZF,ZS,X1,X2,X3,XP,Y1,Y2,Y3,YP
      DOUBLE PRECISION, PARAMETER :: TOPX = 677133.5D0
      DOUBLE PRECISION, PARAMETER :: TOPY = 5462322.5D0

      IF(LT.EQ.0) THEN
            WRITE(LU,*) 'ADDING PARTICLES: ',ROWS,COLS
            DO ROW=0, ROWS-1
                  DO COL=0, COLS-1
                        DO IELEM=1,NELEM
                  !           FIRST 3 POINTS OF ELEMENTS ON BOTTOM IN 3D COINCIDE
                  !           WITH POINTS IN 2D
                              XP = TOPX + 0.5D0 + COL
                              YP = TOPY-ROW-0.5D0
                              N1=MESH%IKLE%I(IELEM         )
                              N2=MESH%IKLE%I(IELEM+  NELMAX)
                              N3=MESH%IKLE%I(IELEM+2*NELMAX)
                              X1=MESH%X%R(N1)
                              X2=MESH%X%R(N2)
                              X3=MESH%X%R(N3)
                              Y1=MESH%Y%R(N1)
                              Y2=MESH%Y%R(N2)
                              Y3=MESH%Y%R(N3)
                              DET1=(X3-X2)*(YP-Y2)-(Y3-Y2)*(XP-X2)
                              DET2=(X1-X3)*(YP-Y3)-(Y1-Y3)*(XP-X3)
                              DET3=(X2-X1)*(YP-Y1)-(Y2-Y1)*(XP-X1)
                              IF(DET1.GE.-EPSILO.AND.
     &                           DET2.GE.-EPSILO.AND.
     &                           DET3.GE.-EPSILO) THEN 
                                    CALL ADD_PARTICLE(XP,YP,0.D0,(ROW)*COLS+COL,1,NFLOT,
     &                                                NFLOT_MAX,XFLOT,YFLOT,YFLOT,TAGFLO,CLSFLO,
     &                                                SHPFLO,SHPFLO,ELTFLO,ELTFLO,MESH,1,
     &                                                0.D0,0.D0,0.D0,0.D0,0,0)
                                    GOTO 30
                              ENDIF
                        ENDDO

30                      CONTINUE
                  ENDDO
            ENDDO
      ENDIF

      RETURN
      END
/*COMPUTATION ENVIRONMENT*
PARALLEL PROCESSORS = 4
GEOMETRY FILE = 'mesh.slf'
GEOMETRY FILE FORMAT = 'SERAFIND'
BOUNDARY CONDITIONS FILE = 'T100D60.cli'
RESULTS FILE = 'T100D60.slf'
RESULTS FILE FORMAT = 'SERAFIND'
FORTRAN FILE = 'user_flot.f'
/*OUTPUT*
TITLE = 'TG14, T100D60 DROGUES'
VARIABLES FOR GRAPHIC PRINTOUTS = 'U,V,H,S,R'
NUMBER OF PRIVATE ARRAYS = 1
GRAPHIC PRINTOUT PERIOD = 300
LISTING PRINTOUT PERIOD = 600
MASS-BALANCE = YES
INFORMATION ABOUT SOLVER = YES
DEBUGGER = 0
/*INITIAL CONDITIONS*
/*BOUNDARY CONDITIONS*
/*PHYSICAL PARAMETERS*
RAINFALL-RUNOFF MODEL = 0
LAW OF BOTTOM FRICTION = 4
FRICTION COEFFICIENT = 0
RAIN OR EVAPORATION = YES
DURATION OF RAIN OR EVAPORATION IN HOURS = 5
RAIN OR EVAPORATION IN MM PER DAY = 1101.6
/*NUMERICAL PARAMETERS, SOLVER*
EQUATIONS = 'SAINT-VENANT FV'
FINITE VOLUME SCHEME = 5
SOLVER ACCURACY = 1.E-6
VARIABLE TIME-STEP = YES
DESIRED COURANT NUMBER = 0.9
ORIGINAL DATE OF TIME = 2000;01;01
ORIGINAL HOUR OF TIME = 00;00;00
DURATION = 14400

ASCII DROGUES FILE        : flo.dat
/DROGUES FILE FORMAT       : BKBINPCL
/BINARY DROGUES FILE       : flo.pcl

MAXIMUM NUMBER OF DROGUES                    : 26415
PRINTOUT PERIOD FOR DROGUES                  : 1000
STOCHASTIC DIFFUSION MODEL = 0
The administrator has disabled public write access.

Unable to run particle tracking in parallel 1 year 2 months ago #43108

  • pham
  • pham's Avatar
  • OFFLINE
  • Administrator
  • Posts: 1549
  • Thank you received: 598
Hello,

Which release do you use?
There is a recent bug fixed for drogues last January just after the release of v8p4r0.
Please find enclosed the modified file.

Can you try to replace it and tell if the problem still occurs please?

Hope this helps,

Chi-Tuan

File Attachment:

File Name: streamline.f
File Size: 379 KB
The administrator has disabled public write access.

Unable to run particle tracking in parallel 1 year 2 months ago #43126

  • N_Strahl
  • N_Strahl's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 12
Thanks for the response. Unfortunately, I get the same exact error when adding the most recent version of streamline.f. The error occurs in the ADD_PARTICLE subroutine. I am guessing some particles are at the partition boundaries. I was thinking It might be possible to remove or ignore the particle IF(NEWTOT-OLDTOT.GT.1) in line 8904. Any idea if this might work? Thanks in advance.
The administrator has disabled public write access.
Moderators: pham

The open TELEMAC-MASCARET template for Joomla!2.5, the HTML 4 version.