1

This was a question that was spawned from a previous question (see Downloading a LOT of files using Python) but was so much more specific than what I originally asked that I thought it deserved its own question.

When running python multiprocessing if I try to download a batch of files at once using threading it throws as error on only some of the files. This is the error, obviously there is an error with urllib2 opening the file but the question is, why?

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
urllib2.URLError: <urlopen error ftp error: >

What is weird is that if I download the files one at a time I do not get this error. And the error is (normally) not consistent. If I run the same process twice it will throw the same error but on different files. This leads me to think that the problem is the interaction of the threads. Maybe 2 threads are trying to ping the site at the same time? Does anyone know what might be causing this?

The machine I am using is a LinuxBox running RedHat with 32 cores.

Here is the code I am using:

from __future__ import division
import pandas as pd
import numpy as np
import urllib2
import os
import linecache
from multiprocessing import Pool
import time

#making our list of urls to download from
data2=pd.read_csv("edgar14A14C.csv")

flist=np.array(data2['filename'])
print len(flist)
print flist

os.chdir(str(os.getcwd())+str('/edgar14A14C'))

###below we have a script to download all of the files in the data2 database
###here you will need to create a new directory named edgar14A14C in your CWD

def job(url):
    print "I'm doing something!"
    file_name = str(url.split('/')[-1])
    u = urllib2.urlopen(url)
    f = open(file_name, 'wb')
    f.write(u.read())
    print file_name
    f.close()


urls = ["ftp://ftp.sec.gov/{0:s}".format(f) for f in flist]
pool = Pool(processes=20)
pool.map(job, urls)

1 Answer 1

0

Processes are limited to only having a specific number of file pointers open at a given time. urllib2.urlopen(url) opens a file pointer (the socket). After you are done with the data, be sure to close it: u.close()

3
  • These can also be called file descriptors.
    – Nathaniel
    Jun 25, 2014 at 21:14
  • I just tried adding in a u.close() command and I still end up getting the same errors. Does it seem like I am doing anything else wrong with the memory? Is is maybe executing f.close() before it has a chance to read or write the file?
    – sfortney
    Jun 25, 2014 at 21:38
  • It could be something on the other end preventing you from download so many files in such a short time. You could try catching the exception and logging it to see how many occur per run.
    – Nathaniel
    Jun 25, 2014 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.