21

The problem: I need to send many HTTP requests to a server. I can only use one connection (non-negotiable server limit). The server's response time plus the network latency is too high – I'm falling behind.

The requests typically don't change server state and don't depend on the previous request's response. So my idea is to simply send them on top of each other, enqueue the response objects, and depend on the Content-Length: of the incoming responses to feed incoming replies to the next-waiting response object. In other words: Pipeline the requests to the server.

This is of course not entirely safe (any reply without Content-Length: means trouble), but I don't care -- in that case I can always retry any queued requests. (The safe way would be to wait for the header before sending the next bit. That'd might help me enough. No way to test beforehand.)

So, ideally I want the following client code (which uses client delays to mimic network latency) to run in three seconds.

Now for the $64000 question: Is there a Python library which already does this, or do I need to roll my own? My code uses gevent; I could use Twisted if necessary, but Twisted's standard connection pool does not support pipelined requests. I also could write a wrapper for some C library if necessary, but I'd prefer native code.

#!/usr/bin/python

import gevent.pool
from gevent import sleep
from time import time

from geventhttpclient import HTTPClient

url = 'http://local_server/100k_of_lorem_ipsum.txt'
http = HTTPClient.from_url(url, concurrency=1)

def get_it(http):
    print time(),"Queueing request"
    response = http.get(url)
    print time(),"Expect header data"
    # Do something with the header, just to make sure that it has arrived
    # (the greenlet should block until then)
    assert response.status_code == 200
    assert response["content-length"] > 0
    for h in response.items():
        pass

    print time(),"Wait before reading body data"
    # Now I can read the body. The library should send at
    # least one new HTTP request during this time.
    sleep(2)
    print time(),"Reading body data"
    while response.read(10000):
        pass
    print time(),"Processing my response"
    # The next request should definitely be transmitted NOW.
    sleep(1)
    print time(),"Done"

# Run parallel requests
pool = gevent.pool.Pool(3)
for i in range(3):
    pool.spawn(get_it, http)

pool.join()
http.close()
7
  • Note: as for C libraries, I already found serf at code.google.com/p/serf . Writing working Python bindings for that library unfortunately is not something I'm currently paid for. :-/ Commented Oct 11, 2013 at 8:35
  • Your code looks a bit like what grequests does. Have you taken a look to it? If you have, could you explain why it is not a good fit? (I probably haven't fully, fully understood the question)
    – Savir
    Commented Jul 3, 2015 at 14:15
  • 3
    grequests is a simple async wrapper for requests, i.e. one thread per request, and each is still a send/receive/send-next-bit lockstep that uses its own connection. I need something which opens a single TCP connection and then spawns one thread that sends the request headers, and another which receives the responses and associates them with "their" requests. Commented Aug 6, 2015 at 21:43
  • this example ( code.activestate.com/recipes/576673-python-http-pipelining ) works well. But only in Python2 - if it's converted to Python3 it works only in like 50% cases...
    – MacHala
    Commented Sep 13, 2017 at 8:09
  • A more fun thing to do is to get your server to support http/2, then use something like grpc or hyper.
    – James Lim
    Commented Dec 15, 2017 at 2:39

3 Answers 3

4

Dugong is an HTTP/1.1-only client which claims to support real HTTP/1.1 pipelining. The tutorial includes several examples on how to use it, including one using threads and another using asyncio.

Be sure to verify that the server you're communicating with actually supports HTTP/1.1 pipelining—some servers claim to support HTTP/1.1 but don't implement pipelining.

0
1

I think txrequests could get you most of what you are looking for, using the background_callback to en-queue processing of responses on a separate thread. Each request would still be it's own thread but using a session means by default it would reuse the same connection.

https://github.com/tardyp/txrequests#working-in-the-background

1
  • Note that txrequests is actually an unmaintained fork of the more actively developed requests-futures with the main changes being the use of twisted's thread pools instead of Python's. Not clear to me what the intended benefit of that is.
    – Seb
    Commented Sep 28, 2023 at 13:38
-4

It seems you are running python2.

For python3 >= 3.5 you could use async/await loop See asyncio

Also, there is a library built on top for better, easier use called Trio, available on pip.


Another thing I can think of is multiple threads with locks. I will think on how to better explain this or could it even work.

3
  • 1
    Please explain which async HTTP library is able to queue more than one request onto a single connection. aiohttp can't do it. Commented Aug 17, 2017 at 20:23
  • the anser has nothing to do with pipelining
    – MacHala
    Commented Sep 13, 2017 at 8:08
  • aiohttp might add requests pipelining in ClientSession: github.com/aio-libs/aiohttp/issues/1740
    – DurandA
    Commented Oct 12, 2017 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.