From python threading documentation
In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.
Now I have a thread worker like this
def worker(queue):
queue_full = True
while queue_full:
try:
url = queue.get(False)
w = Wappalyzer(url)
w.analyze()
queue.task_done()
except Queue.Empty:
queue_full = False
Here w.analyze()
doing two things
- Scrape the url using
requests
library - Analyzing the scraped html using
pyv8
javascript library
As far as I know, 1
is I/O bound and 2
is CPU bound.
Does that mean, GIL applied for 2
and my program won't work properly?
requests
is CPU bound, or at least it locks its thread until the request completes. For a callback-capable library, I would check out requests-futures.requests
(andurllib
,httplib2
, etc) are all very much I/O bound.threading
speeds all of them up.sleep(1000)
into a thread the thread will be blocked for some time, but it won't do any work and will release the GIL in between. Same goes for any other kind of IO request.