open TELEMAC-MASCARET The mathematically superior suite of solvers

Hi everyone,

I am using Telemac2D version v8p4r0 for storm surge modelling. I created have binary atmospheric data file which contains, mean sea level pressure and wind U and V velocities. This file is written to Selafin format, which matches the mesh.

Could anyone please provide me with the normal operating procedure for partitioning binary atmospheric file in Telemac2D? Specifically, I would like to know if the code should be writing the atmospheric file partitions in parallel. At present it appears to write each partitioned file one after the other and it taken approximately 5-hours to complete this task. Once it has completed writing the partition atmospheric files the model starts and runs in parallel fine. So, I’m wondering if I missed something when I compiled the code.

Thank you in advance for your help!

Best regards,

Hello Greeve,

Just a quick reply, with the same mesh containing the atmospheric data, it took me about 5 minutes or less (in a cluster with 50+ cores) to do the partition job before the calculations. The size of my atmospheric file is about 1G.

Maybe you can check the machine you're using, or check the libraries you're now using ? I'm not an expert, just FYI.

Hope it helps,
Grebe

Hello Glen,

Can you provide:
- your steering file,
- run command run_telfile.py scan name_of_binary_atmo_data_file and copy and paste what is written here please.

If you use the keyword BINARY ATMOSPHERIC DATA FILE, as written in the dictionary for SUBMIT line with the last item SERAFIN, that means this file is partitioned like a result file, a previous computation file.
If there are a lot of records in the BINARY ATMOSPHERIC DATA FILE, that may take a lot of time to partition it with PARTEL (same issue when merging file with GRETEL).
This step is done in sequential. There used to be a parallel version of PARTEL but I do not know if it really work at the moment.

Anyway, one workaround if you do not change input files so much is to do the partition only once (with option --split and by giving a temporary folder name with option -w, see the help with telemac2d.py -h).
Then if you have Fortran file, run telemac2d.py -x
Then always run in temporary file with --run option (you then skip the partition step). You will have to merge your self the split results file.

Hope this helps,

Chi-Tuan

Hi Chi-Tuan,

For anyone interested, a colleague and I have finally had some time to dig into the partitioning of the binary atmosphere file a bit more. The “partel” partitioning tool processes each partition sequentially and re-reads the entire input dataset from disk in every instance. This leads to very long runtimes when the overall number of partitions is large (hundreds). A straightforward optimisation in “parres.f” allocates a very large array that holds the entire input data file in RAM; it is populated when the first partition is processed. This approach minimises code changes in parres.f.
[…]
DOUBLE PRECISION,ALLOCATABLE :: VAL(:),VAL_INP(:)
! VERY LARGE MEMORY BUFFER TO AVOID RE-READING INPUT DATA
DOUBLE PRECISION,ALLOCATABLE :: INPUTBUF(:,:,:)
! GEOMETRY INFORMATION
INTEGER NPOIN_GEO,TYP_ELM_GEO,NELEM_GEO,NPTFR_GEO,NPTIR_GEO,
& NDP_GEO,NPLAN_GEO
[…]
ALLOCATE(VAL(NPOIN_P),STAT=IERR)
CALL CHECK_ALLOCATE(IERR,'PARRES:VAL')

! FIRST PARTITION: ALLOCATE BUFFER FOR INPUT DATA
IF ( IPART .EQ. 1 ) THEN
WRITE(LU,*) 'ALLOCATING LARGE MEMORY BUFFER WITH',
& NPOIN_INP*NVAR_INP*NTIMESTEP,
& 'ELEMENTS'
ALLOCATE(INPUTBUF(NPOIN_INP, NVAR_INP, NTIMESTEP), STAT=IERR)
CALL CHECK_ALLOCATE(IERR,'PARRES:INPUTBUF')
END IF

! LOOPING ON THE TIMESTEP AND VARIABLE OF INP FILE
DO ITIME=1,NTIMESTEP
CALL GET_DATA_TIME(INPFORMAT,NINP,ITIME-1,TIMES,IERR)
CALL CHECK_CALL(IERR,'PARTEL:GET_DATA_TIME:NINP')
WRITE(LU,*) ' -- WRITING TIMESTEP',ITIME-1,' AT',REAL(TIMES)
! Loop on all the variables
DO IVAR=1,NVAR_INP

! POPULATE BUFFER
IF ( IPART .EQ. 1 ) THEN
CALL GET_DATA_VALUE(INPFORMAT,NINP,ITIME-1,
& VARLIST(IVAR)(1:16),VAL_INP,
& NPOIN_INP,IERR)
INPUTBUF(:, IVAR, ITIME) = VAL_INP(:)
ENDIF

! GETTING THE VALUE NEEDED FOR THAT PARTITION
IF(NPLAN_INP.GT.1) THEN
DO I=1,NPOIN_P
VAL(I) = INPUTBUF(KNOLG3D(I), IVAR, ITIME)
ENDDO
ELSE
DO I=1,NPOIN_P
VAL(I) = INPUTBUF(KNOLG(I), IVAR, ITIME)
ENDDO
ENDIF

CALL ADD_DATA(INPFORMAT,NINP_PAR,VARLIST(IVAR),TIMES,
& ITIME-1,IVAR.EQ.1,VAL,NPOIN_P,IERR)
CALL CHECK_CALL(IERR,'PARRES:ADD_DATA:NINP_PAR')
ENDDO
ENDDO

The runtime machine will evidently need to have sufficiently large memory in case of high-resolution meteorology files with long timeseries, but the speedup is significant.

For example, running partel in its original and optimised versions to split an 1.4 GiB atmospheric data file holding 48 timesteps with temperature, pressure, and two wind component fields on 1,842,047 mesh nodes and 3,614,079 cells (triangles) into 200 partitions resulted in the following runtimes:
• Original version: 1259s
• Optimised version: 20s (x63 speed-up)

Both versions produced bit-identical output files. The optimised version required around 8 Bytes x 1842047 mesh nodes x 4 fields x 48 timesteps = 2.6 GiB of memory - around twice the size of the atmospheric data file, as the latter stores data in single precision format, while partel operates in double precision internally. A more elaborate implementation of the data caching mechanism would avoid this doubling of RAM requirements, at the expense of having to implement more significant code changes.

Hello Glen,

Thank you very much for you investigation and suggestions. I have to discuss with at least a colleague of mine.
It could be a 2nd option to partition files as there are 2 methods to gather split files with gretel.

Chi-Tuan