2

So I got a simple runner script that will use Popen to call the spider. The runner script is as follows:

from subprocess import Popen
import time
def runSpider():
    p = Popen(["scrapy", "crawl", "spider1"], 
        cwd="C:\Users\Kasutaja\Desktop\scrapyTest")
    stdout, stderr = p.communicate()
    time.sleep(15)

runSpider()

The directory is like this:

 -----scrapyTest: 
 --------------------scrapyTest[folder]: spider[folder], items.py, pipelines.py, settings.py
 --------------------runner.py
 --------------------scrapy.cfg

The spider runs perfectly from the dir: C:\Users\Kasutaja\Desktop\scrapyTest when I run it from the cmd line.

When I run my runner.py script I get:

The system cannot find the file specified

EDIT:

After changing the Popen to this:

p = Popen(["C:\Users\Kasutaja\Desktop\scrapyTest","scrapy", "crawl", "spider1"])

I get the error:

    C:\Users\Kasutaja\Desktop\scrapyTest>python runner.py
Traceback (most recent call last):
  File "runner.py", line 13, in <module>
    runSpider()
  File "runner.py", line 8, in runSpider
    p = Popen(["C:\Users\Kasutaja\Desktop\scrapyTest","scrapy", "crawl", "spider
1"])
  File "C:\Python27\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "C:\Python27\lib\subprocess.py", line 896, in _execute_child
    startupinfo)
WindowsError: [Error 5] Access is denied

If it matters, I have admin rights. I have also tried now, running the script with cmd specifically opened from start menu and with admin rights, but still get the same error.

3
  • Could you provide a full traceback?
    – alecxe
    Jun 27, 2013 at 11:54
  • What happens if you put the runner.py script into the "C:\Users\Kasutaja\Desktop\scrapyTest" and run it from there?
    – Talvalin
    Jun 27, 2013 at 15:50
  • The script is there, as when I call the scrapy crawl spider1 command from cmd-line also from there, it works. just not by calling it through runner.py
    – fusi0n
    Jun 27, 2013 at 15:54

1 Answer 1

0

From the docs

If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd. (Emphasis mine)

to me that means you have to do something like

Popen(["C:\Users\Kasutaja\Desktop\scrapyTest\scrapy", "crawl", "spider1"])

this will execute a program scrapy with arguments crawl and spider1

1
  • Thank you, but now I get Windows Error[5]: access is denied
    – fusi0n
    Jun 27, 2013 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.