50

I have a site that runs with follow configuration:

Django + mod-wsgi + apache

In one of user's request, I send another HTTP request to another service, and solve this by httplib library of python.

But sometimes this service don't get answer too long, and timeout for httplib doesn't work. So I creating thread, in this thread I send request to service, and join it after 20 sec (20 sec - is a timeout of request). This is how it works:

class HttpGetTimeOut(threading.Thread):
    def __init__(self,**kwargs):
        self.config = kwargs
        self.resp_data = None
        self.exception = None
        super(HttpGetTimeOut,self).__init__()
    def run(self):

        h = httplib.HTTPSConnection(self.config['server'])
        h.connect()
        sended_data = self.config['sended_data']
        h.putrequest("POST", self.config['path'])
        h.putheader("Content-Length", str(len(sended_data)))
        h.putheader("Content-Type", 'text/xml; charset="utf-8"')
        if 'base_auth' in self.config:
            base64string = base64.encodestring('%s:%s' % self.config['base_auth'])[:-1]
            h.putheader("Authorization", "Basic %s" % base64string)
        h.endheaders()

        try:
            h.send(sended_data)
            self.resp_data = h.getresponse()
        except httplib.HTTPException,e:
            self.exception = e
        except Exception,e:
            self.exception = e

something like this...

And use it by this function:

getting = HttpGetTimeOut(**req_config)
getting.start()
getting.join(COOPERATION_TIMEOUT)
if getting.isAlive(): #maybe need some block
    getting._Thread__stop()
    raise ValueError('Timeout')
else:
    if getting.resp_data:
        r = getting.resp_data
    else:
        if getting.exception:
            raise ValueError('REquest Exception')
        else:
            raise ValueError('Undefined exception')

And all works fine, but sometime I start catching this exception:

error: can't start new thread

at the line of starting new thread:

getting.start()

and the next and the final line of traceback is

File "/usr/lib/python2.5/threading.py", line 440, in start
    _start_new_thread(self.__bootstrap, ())

And the answer is: What's happen?

Thank's for all, and sorry for my pure English. :)

10 Answers 10

47

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

You should probably look at the number of threads you're creating; the maximum number you will be able to create will be determined by your environment, but it should be in the order of hundreds at least.

It would probably be a good idea to re-think your architecture here; seeing as this is running asynchronously anyhow, perhaps you could use a pool of threads to fetch resources from another site instead of always starting up a thread for every request.

Another improvement to consider is your use of Thread.join and Thread.stop; this would probably be better accomplished by providing a timeout value to the constructor of HTTPSConnection.

2
  • 23
    Note that the number of threads running can be shown using threading.active_count().
    – 101
    Commented Jun 7, 2016 at 4:13
  • "Almost certainly" maybe, but I had the problem with "can't start new thread" because of insufficient memory available. I limited available memory for a test where I want to provoke MemoryError. Unfortunately the amount of memory required depends on the Python version, so my workstation happily started the thread and later encountered a MemoryError, while the Jenkins CI failed with "can't start new thread".
    – peschü
    Commented Dec 27, 2021 at 10:57
14

You are starting more threads than can be handled by your system. There is a limit to the number of threads that can be active for one process.

Your application is starting threads faster than the threads are running to completion. If you need to start many threads you need to do it in a more controlled manner I would suggest using a thread pool.

13

I was running on a similar situation, but my process needed a lot of threads running to take care of a lot of connections.

I counted the number of threads with the command:

ps -fLu user | wc -l

It displayed 4098.

I switched to the user and looked to system limits:

sudo -u myuser -s /bin/bash

ulimit -u

Got 4096 as response.

So, I edited /etc/security/limits.d/30-myuser.conf and added the lines:

myuser hard nproc 16384

myuser soft nproc 16384

Restarted the service and now it's running with 7017 threads.

Ps. I have a 32 cores server and I'm handling 18k simultaneous connections with this configuration.

2
  • 1
    thegeekdiary.com/… This will help you to understand how /etc/security/limits.d/ conf files work
    – subha.py
    Commented Jul 23, 2021 at 10:20
  • Is ok to manually edit "30-<myuser>.conf" file? Is there any step before this?
    – Cloud Cho
    Commented Jul 28 at 7:45
6

I think the best way in your case is to set socket timeout instead of spawning thread:

h = httplib.HTTPSConnection(self.config['server'], 
                            timeout=self.config['timeout'])

Also you can set global default timeout with socket.setdefaulttimeout() function.

Update: See answers to Is there any way to kill a Thread in Python? question (there are several quite informative) to understand why. Thread.__stop() doesn't terminate thread, but rather set internal flag so that it's considered already stopped.

0
6

I found this question because pip failed to install packages while inside a docker container. A related issue on the pip repo suggests this to be a poorly worded exception coming from rich that is thrown when the system reaches the limit for the maximum number of threads for some reason. The following fixes are given:

  • Upgrading Docker to a version > 20.10.7
  • Running pip with -q to suppress the rich output
1
  • adding the -q worked for me on ubuntu 20.04 trying to follow the catnip example from the intro docker tutorial. then the example didn't work but possibly due to poorly setup requirements.
    – rrosa
    Commented Oct 25, 2023 at 4:04
5

I completely rewrite code from httplib to pycurl.

c = pycurl.Curl()
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.CONNECTTIMEOUT, CONNECTION_TIMEOUT)
c.setopt(pycurl.TIMEOUT, COOPERATION_TIMEOUT)
c.setopt(pycurl.NOSIGNAL, 1)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.SSL_VERIFYHOST, 0)
c.setopt(pycurl.SSL_VERIFYPEER, 0)
c.setopt(pycurl.URL, "https://"+server+path)
c.setopt(pycurl.POSTFIELDS,sended_data)

b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)

c.perform()

something like that.

And I testing it now. Thanks all of you for help.

4

If you are tying to set timeout why don't you use urllib2.

3
  • urllib2 doesn't has connection time out.
    – Oduvan
    Commented Dec 3, 2009 at 18:39
  • 1
    urllib2 does have timeout. <snip> urllib2.urlopen(url[, data][, timeout])</snip>
    – piyer
    Commented Dec 4, 2009 at 4:02
  • 1
    timeout argument is new in Python 2.6 Commented Dec 4, 2009 at 7:43
3

I'm running a python script on my machine only to copy and convert some files from one format to another, I want to maximize the number of running threads to finish as quickly as possible.

Note: It is not a good workaround from an architecture perspective If you aren't using it for a quick script on a specific machine.

In my case, I checked the max number of running threads that my machine can run before I got the error, It was 150

I added this code before starting a new thread. which checks if the max limit of running threads is reached then the app will wait until some of the running threads finish, then it will start new threads

while threading.active_count()>150 :
    time.sleep(5)
mythread.start()
2
  • How did you check and discover the 150 value?
    – jtlz2
    Commented Dec 1, 2023 at 9:15
  • 1
    I kept printing the number of threading.active_count() before I start a new thread, then I found the error happens after 150, in my case. The is not the best answer, But I wanted to finish the task without spending a lot of time to enhance the script code. Commented Dec 15, 2023 at 9:09
1

If you are using a ThreadPoolExecutor, the problem may be that your max_workers is higher than the threads allowed by your OS.

It seems that the executor keeps the information of the last executed threads in the process table, even if the threads are already done. This means that when your application has been running for a long time, eventually it will register in the process table as many threads as ThreadPoolExecutor.max_workers

2
  • And the solution is ?
    – Yahya
    Commented Sep 16, 2023 at 22:31
  • if you are using ThreadPoolExecutor, reduce max_workers, e.g. with futures.ThreadPoolExecutor(max_workers=10) as exc: .... Commented Apr 2 at 21:44
0

As far as I can tell it's not a python problem. Your system somehow cannot create another thread (I had the same problem and couldn't start htop on another cli via ssh).

The answer of Fernando Ulisses dos Santos is really good. I just want to add, that there are other tools limiting the number of processes and memory usage "from the outside". It's pretty common for virtual servers. Starting point is the interface of your vendor or you might have luck finding some information in files like /proc/user_beancounters

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.