Welcome, Guest
Username: Password: Remember me
Forgot your password? Forgot your username? Create an account
Forum
PRE/POST PROCESSORS
Scripts
Benchmark performance of ppUtils vs Selafin vs TelemacFile
Flat Threaded

TOPIC: Benchmark performance of ppUtils vs Selafin vs TelemacFile

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44231

  • tomsail
  • tomsail's Avatar
  • OFFLINE
  • Junior Boarder
  • Posts: 43
  • Thank you received: 17
Thanks Nico.

Seeing these results I definitely need to install it in my conda package too.
Would you know how it is more than 20 times faster?

For the xarray implementation, I think the most important is to have light supporting scripts and dependencies.

The processing time (for reading/writing input/output files) can then be saved with paralleling, dask arrays and lazy loading.

Here is the file if you wish to complete my benchmark :)
The administrator has disabled public write access.
The following user(s) said Thank You: sebourban

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44232

  • nicogodet
  • nicogodet's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 157
  • Thank you received: 39
Seeing these results I definitely need to install it in my conda package too.
It's quite easy to compile it ;)
Would you know how it is more than 20 times faster?
I'm not a expert but it is probably due to python delegating the reading to the compiled FORTRAN backend.
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44235

  • nicogodet
  • nicogodet's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 157
  • Thank you received: 39
With your file
Time taken by ppUtils: 6.4874138832092285 seconds
Time taken by ppUtils_access: 2.476961135864258 seconds
Time taken by selafin: 1.0657806396484375 seconds
Time taken by selafin_access: 0.3386349678039551 seconds
Time taken by telemacHermes: 0.14725613594055176 seconds
Time taken by telemacHermes_access: 0.031247615814208984 seconds
Time taken by pyTelTools: 0.9454250335693359 seconds
Time taken by pyTelTools_access: 1.0548181533813477 seconds
The administrator has disabled public write access.
The following user(s) said Thank You: Lux

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44237

  • Lux
  • Lux's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 96
  • Thank you received: 39
Thank you for benchmark and sorry for the delay but for PyTelTools, the reading a record can be speedup in not reading again the header (the presence of 2 "with" is not optimal).

Here is a code :
# PyTelTools
    @timer
    def pyTelTools(f):
        resin = Serafin.Read(f, 'en')
        resin.__enter__()
        resin.read_header()
        return resin

    @timer
    def pyTelTools_access(slf, i_record, f):
        slf.read_var_in_frame(i_record, slf.header.header.var_IDs[0])

    file_3d = '/home/tomsail/Documents/work/python/pyPoseidon/Tutorial/test/hindcast/201303/input_wind_tmp.slf'
    resin = pyTelTools(file_3d)
    pyTelTools_access(resin, 10, file_3d)

Hope it helps,
Luc
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44238

  • nicogodet
  • nicogodet's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 157
  • Thank you received: 39
Thomas' file :
Time taken by pyTelTools: 1.1253736019134521 seconds
Time taken by pyTelTools_access: 0.13963556289672852 seconds

Mine:
Time taken by pyTelTools: 0.44992566108703613 seconds
Time taken by pyTelTools_access: 0.055106163024902344 seconds
The administrator has disabled public write access.
The following user(s) said Thank You: sebourban, Lux, tomsail

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44239

  • Lux
  • Lux's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 96
  • Thank you received: 39
Thank you Nicolas for the integration of PyTelTools optimization in the comparison.
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 3 days ago #44241

  • sebourban
  • sebourban's Avatar
  • OFFLINE
  • Administrator
  • Principal Scientist
  • Posts: 814
  • Thank you received: 219
.. another conclusion to this benchmark (although it dependens on compilation and hardware), the Windows version seems just as fast as the linux version.

Good work to all !
Thank you.
Sébastien.
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 2 days ago #44242

  • Lux
  • Lux's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 96
  • Thank you received: 39
Sorry Nicolas, I just realized that all variables should be read for the access benchmark.
But in the script shared initially, PyTelTools was forced to read only the first variable (so Z probably).

If you want to compare same tasks, could you please update PyTelTools (to have commit f6af747) and use read_vars_in_frame (or eventually read_vars_in_frame_as_3d) instead of read_var_in_frame.

Here is a snippet example:
    # PyTelTools
    @timer
    def pyTelTools(f):
        resin = Serafin.Read(f, 'en')
        resin.__enter__()
        resin.read_header()
        return resin

    @timer
    def pyTelTools_access(slf, i_record):
        slf.read_vars_in_frame(i_record)

    file_3d = '/home/tomsail/Documents/work/python/pyPoseidon/Tutorial/test/hindcast/201303/input_wind_tmp.slf'
    resin = pyTelTools(file_3d)
    pyTelTools_access(resin, 10, file_3d)

The output should have the shape: (number of variables, number of nodes).

Remark: the reading in PyTelTools has not been optimized to read simultaneously all variables. But it could be easy optimized while providing also other interfaces (read specific variables).


Best Regards,
Luc
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 2 days ago #44243

  • Lux
  • Lux's Avatar
  • OFFLINE
  • Senior Boarder
  • Posts: 96
  • Thank you received: 39
(I can not edit my previous post, so I post a fix...)

Sorry again, resin.get_time() has be forgotten :
# PyTelTools
    @timer
    def pyTelTools(f):
        resin = Serafin.Read(f, 'en')
        resin.__enter__()
        resin.read_header()
        resin.get_time()
        return resin

    @timer
    def pyTelTools_access(slf, i_record):
        slf.read_vars_in_frame(i_record)

    file_3d = '/home/tomsail/Documents/work/python/pyPoseidon/Tutorial/test/hindcast/201303/input_wind_tmp.slf'
    resin = pyTelTools(file_3d)
    pyTelTools_access(resin, 10, file_3d)
The administrator has disabled public write access.

Benchmark performance of ppUtils vs Selafin vs TelemacFile 9 months 2 days ago #44246

  • nicogodet
  • nicogodet's Avatar
  • OFFLINE
  • Expert Boarder
  • Posts: 157
  • Thank you received: 39
My file
Time taken by pyTelTools: 0.465364933013916 seconds
Time taken by pyTelTools_access: 0.5686259269714355 seconds

Thomas' file
Time taken by pyTelTools: 1.167872667312622 seconds
Time taken by pyTelTools_access: 0.3161137104034424 seconds
The administrator has disabled public write access.
Flat Threaded
Moderators: borisb
Forum
PRE/POST PROCESSORS
Scripts
Benchmark performance of ppUtils vs Selafin vs TelemacFile

The open TELEMAC-MASCARET template for Joomla!2.5, the HTML 4 version.