5

I am attempting to call a python script from a master script. I need the dataframe to be generated only one from within the master script and then passed on to the subprocess script as an argument to be used inside the subprocess.

Following is my attempt at writing the required python master script.

from subprocess import PIPE, Popen
import pandas as pd

test_dataframe = pd.read_excel(r'C:\test_location\file.xlsx',sheetname='Table')

sp = Popen(["python.exe",'C:/capture/test.py'], shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
sp.communicate(test_dataframe)

And here is the error: TypeError: argument 1 must be convertible to a buffer, not DataFrame

This is my first time trying to use the subprocess module so i am not very good at it yet. Any help will be much appreciated.

2 Answers 2

5

Here is a complete example for Python 3.6 of two-way communication between the master script and a subprocess.

master.py

import pandas as pd
import pickle
import subprocess

df = pd.read_excel(r'C:\test_location\file.xlsx',sheetname='Table')

result = subprocess.run(['python', 'call_model.py'], input=pickle.dumps(df), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
returned_df = pickle.loads(result.stdout)
assert df == returned_df

If there is a problem, you can check result.stderr.

subroutine.py

import pickle
import sys

data = pickle.loads(sys.stdin.buffer.read())
sys.stdout.buffer.write(pickle.dumps(data))
3

Subprocess launches another application. The ways that processes may communicate between each other significantly differ from ways that functions communicate within python program. You need to pass your DataFrame through a non pythonic environment. So you need to serialize it in-to a text and then deserialize it on other end. For example you can use pickle module and then sp.communicate(pickle.dumps(test_dataframe)) on one end end pickle.loads(sys.stdin.read()) on another. Or you can write your DataFrame as csv and then parse it again. Or you can use any other format.

3
  • 1
    Hi Alex, thank you for the reply. I attempted to use pickle but after going through the master script, it closes itself without giving any error and no command from subprocess script is run at all. Commented Aug 3, 2017 at 19:29
  • Are you sure, that subproess is not executed? May be it is executed, but you do not check if it has finished successfully? Also check output from the process, it is returned by sp.comminucate. Beside that, check what shell=True does. You are passing command as an array, but also specify shell=True. Commented Aug 3, 2017 at 19:36
  • 1
    My subprocess begins with a print command and ends with raw_input() for me to see whether it runs or not, the pop up window doesn't show anything before closing by itself. I tried with and without using shell, i dont get anything from subprocess either way. I tried running without stdin, stdout and stderr, only then the first print command from subprocess runs but hangs at pickle.loads(sys.stdin.read()) forever. Commented Aug 3, 2017 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.