TELEMAC-MASCARET Forum: run_telfile.py merge with huge file eats all RAM and crash (1/1)

Welcome, Guest

TOPIC: run_telfile.py merge with huge file eats all RAM and crash

run_telfile.py merge with huge file eats all RAM and crash 8 months 19 hours ago #44463

nicogodet
OFFLINE
Expert Boarder
Posts: 157
Thank you received: 39

Hi,

I tried to merge 2 files :: 11Go and 8,5Go with run_telfile.py merge.

$ run_telfile.py merge --kind=time reestuarisation_part_p1.res reestuarisation_part_p2.res test.res


Interpreting command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  ~> Merging reestuarisation_part_p1.res, reestuarisation_part_p2.res into test.res kind=time
Skipping duplicate time 6566400.0 from reestuarisation_part_p2.res
Killed

I ran this on a machine with 32Go of RAM, it crashes.

I retried on 128Go RAM machine and it uses at maximum 75Go of RAM.
This is huge.

I think run_telfile.py merge could be improved by reading/writing by chunk (few timestep at a time) instead of first reading all files, store them in RAM and finally write to disk.

The administrator has disabled public write access.

run_telfile.py merge with huge file eats all RAM and crash 6 months 4 weeks ago #44638

borisb
OFFLINE
Admin
Posts: 128
Thank you received: 64

Hi Nico,

This is indeed a problem because TelemacFile is not currently designed to handle direct writing to disk without going through a memory buffer, at least for the mesh part. You're not the first to point out this problem and I'm going to try to change this by proposing two file usage modes in the constructor, one enabling direct I/O to disk, the other going through a read-write memory buffer.

The administrator has disabled public write access.

The following user(s) said Thank You: nicogodet

run_telfile.py merge with huge file eats all RAM and crash 6 months 2 weeks ago #44717

borisb OFFLINE Admin Posts: 128 Thank you received: 64	Hi Nico, I managed to fix this issue, you should update and recompile your TELEMAC repository. See gitlab.pam-retd.fr/otm/telemac-mascaret/...490dbe0816c32370df78. This fix will be included in the next V8P5R1 bugfix release (available in a couple of weeks, probably).
	The administrator has disabled public write access. The following user(s) said Thank You: nicogodet