Welcome, Guest
Username: Password: Remember me
Forgot your password? Forgot your username? Create an account
Forum
PRE/POST PROCESSORS
Scripts
run_telfile.py merge with huge file eats all RAM and crash
Flat Threaded

TOPIC: run_telfile.py merge with huge file eats all RAM and crash

run_telfile.py merge with huge file eats all RAM and crash 9 months 1 week ago #44463

  • nicogodet
  • nicogodet's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 166
  • Thank you received: 40
Hi,

I tried to merge 2 files :: 11Go and 8,5Go with run_telfile.py merge.
$ run_telfile.py merge --kind=time reestuarisation_part_p1.res reestuarisation_part_p2.res test.res


Interpreting command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  ~> Merging reestuarisation_part_p1.res, reestuarisation_part_p2.res into test.res kind=time
Skipping duplicate time 6566400.0 from reestuarisation_part_p2.res
Killed
I ran this on a machine with 32Go of RAM, it crashes.

I retried on 128Go RAM machine and it uses at maximum 75Go of RAM.
This is huge.

I think run_telfile.py merge could be improved by reading/writing by chunk (few timestep at a time) instead of first reading all files, store them in RAM and finally write to disk.
The administrator has disabled public write access.

run_telfile.py merge with huge file eats all RAM and crash 8 months 6 days ago #44638

  • borisb
  • borisb's Avatar
  • OFFLINE
  • Admin
  • Posts: 129
  • Thank you received: 67
Hi Nico,

This is indeed a problem because TelemacFile is not currently designed to handle direct writing to disk without going through a memory buffer, at least for the mesh part. You're not the first to point out this problem and I'm going to try to change this by proposing two file usage modes in the constructor, one enabling direct I/O to disk, the other going through a read-write memory buffer.
The administrator has disabled public write access.
The following user(s) said Thank You: nicogodet

run_telfile.py merge with huge file eats all RAM and crash 7 months 3 weeks ago #44717

  • borisb
  • borisb's Avatar
  • OFFLINE
  • Admin
  • Posts: 129
  • Thank you received: 67
Hi Nico,

I managed to fix this issue, you should update and recompile your TELEMAC repository. See gitlab.pam-retd.fr/otm/telemac-mascaret/...490dbe0816c32370df78.

This fix will be included in the next V8P5R1 bugfix release (available in a couple of weeks, probably).
The administrator has disabled public write access.
The following user(s) said Thank You: nicogodet
Flat Threaded
Moderators: borisb
Forum
PRE/POST PROCESSORS
Scripts
run_telfile.py merge with huge file eats all RAM and crash

The open TELEMAC-MASCARET template for Joomla!2.5, the HTML 4 version.