TELEMAC-MASCARET Forum: Changing subprocess argument 'shell=True' to False (1/1)

Changing subprocess argument 'shell=True' to False 4 years 2 months ago #36635

cyamin
OFFLINE
openTELEMAC Guru
Posts: 997
Thank you received: 234

Hello all,
I have installed TELEMAC (for various reasons) on a network drive accessed using UNC paths, which I believe isn't uncommon or forbidden. Being a Windows user, I have being plagued with the native shell's (CMD.exe) limitation to handle UNC paths. So far I had been working around it using registry hacks, but the issue keeps coming back at every major Windows update. So I decided to seek a proper solution.
The obvious one was to use Powershell instead of CMD, which I am already implementing (e.g. job submission and recollection in the HPC Queue). However, I have stumbled across the hidden 'shell=True' parameter in the various calls using python's subprocess module. This argument forces using the CMD shell that presents the UNC limitation leading to a dead end.
Changing subprocess argument 'shell=True' to False, worked for compiling the whole system (including the api) but fails in the partitioning step when calling partel (and possibly more failures are waiting downstream, at least gretel).
I have pinpointed the issue in the issue in messages.py, line 258:

code.value = sp.call(exe, shell=True)

Changing shell=False freezes partel execution with the following being the printout:

+-------------------------------------------------+
   PARTEL/PARRES: TELEMAC METISOLOGIC PARTITIONER

   REBEKKA KOPMANN & JACEK A. JANKOWSKI (BAW)
                  JEAN-MICHEL HERVOUET (LNHE)
                  CHRISTOPHE DENIS     (SINETICS)
                  YOANN AUDOUIN        (LNHE)
   PARTEL (C) COPYRIGHT 2000-2002
   BUNDESANSTALT FUER WASSERBAU, KARLSRUHE

   METIS 5.0.2 (C) COPYRIGHT 2012
   REGENTS OF THE UNIVERSITY OF MINNESOTA

   BIEF TRUNK  (C) COPYRIGHT 2012 EDF
 +-------------------------------------------------+


   MAXIMUM NUMBER OF PARTITIONS:       100000

 +--------------------------------------------------+

 --INPUT FILE NAME <INPUT_NAME>:

I will keep digging, but I have two questions while at it:

Does Linux installations behave better when changing 'shell=False'?
Is there a coding/functionality limitation (that obviously haven't figured out myself), that excludes 'shell' from being set to False?

Any help would be greatly appreciated.

Best Regards,
Costas

The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 2 months ago #36637

cyamin
OFFLINE
openTELEMAC Guru
Posts: 997
Thank you received: 234

Progress update:

According to subprocess documentation:

args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.

So, if setting 'shell=False', the args must be a sequence of strings. Tried to split the command, but it doesn't work for partel. It seems that the <INPUT_FILE> cannot be 'fed' to the executable, probably because the redirection character "<" cannot be interpreted.
This is getting out of my league and it seems I should stick to registry hacks. I hope this issue will be addressed in the future.

Costas

The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 2 months ago #36639

cyamin OFFLINE openTELEMAC Guru Posts: 997 Thank you received: 234	Progress update 2: It is interesting how the process of posting springs new ideas. The way to translate sp.Popen('partel.exe < partel_T3DGEO.par >> partel_T3DGEO.log', shell=True) is sp.Popen('partel.exe', stdin=open('partel_T3DGEO.par'), stdout=open('partel_T3DGEO.log', 'w'), shell=False) This of course means that messages.py and run.py need adaptation, but alas, it is doable. Costas
	The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 2 months ago #36645

yugi OFFLINE openTELEMAC Guru Posts: 851 Thank you received: 244	Hi, Intersting research. I'll try to have a look at it. Thanks for the work.
	There are 10 types of people in the world: those who understand binary, and those who don't. The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 3 weeks ago #37091

cyamin OFFLINE openTELEMAC Guru Posts: 997 Thank you received: 234	Hello Yoann, Is there any chance you can have a look at the issue any time soon? Can I hold my breath or should I hack the registry again? Costas
	The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 3 weeks ago #37095

yugi OFFLINE openTELEMAC Guru Posts: 851 Thank you received: 244	Hi, It is still in my todo list. However it will not be in v8p2 I did not have time to do it before we freeze development for v8p2.
	There are 10 types of people in the world: those who understand binary, and those who don't. The administrator has disabled public write access.

Changing subprocess argument 'shell=True' to False 4 years 3 weeks ago #37097

cyamin OFFLINE openTELEMAC Guru Posts: 997 Thank you received: 234	Hacking it is then. I 'll nag again at the next major windows update.
	The administrator has disabled public write access.