1

Context: I've got a simple python script that writes a bunch of files to s3 (~70-100) every few seconds. Because its I/O bound, I wanted to thread the write process so the script performs better. I'm using threading to build my threads.

Question: Since my threads are a) non-daemons and b) they only have 1 task to perform (write a file), if I loop over my list of threads and call .join() will they finish their task and exit gracefully? Do I even need to call join() here or will they just exit when they're done? I believe join() is the way to go here but since i'm very new to python, I don't know what I don't know....

Here's some simplified code for reference:

buildOutput() #calls the section below 
for thread in threads:
   thread.join()
time.sleep(60)

calls:

  for item in out[fileRoot]: 
        #write individiual files
        key = findKey(item, FILE_KEY)
        full_key = FILE_PATH + str(key) + FILE_TYPE
        t = FileWriter(item, full_key)
        t.start()
        threads.append(t) #global threads list for the script

where FileWriter is the class that does the writing.

1 Answer 1

1

Join makes sure that the main thread will wait until joined thread finishes its task. There is a good ascii art given here.

So you'd better use join when your child threads are performing I/O to prevent any unexpected behavior.

2
  • right but what happens to the thread when its done with its task? are there any mechanisms by which I can be sure that nothing is floating around like a zombie process or garbage collection concerns? something like that?
    – Brad
    Jul 22, 2013 at 17:17
  • treated like any other method, deleted from stack. Since we are talking about threads there won't be anything like a zombie process because as you said they finish their task. However, without join, if the main thread exits while the other haven't finished, their stack will remain (not good).
    – sgun
    Jul 22, 2013 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.