3

Python's BaseHTTPRequestHandler has an issue with forms sent through post!

I have seen other people asking the same question (Why GET method is faster than POST?), but the time difference in my case is too much (1 second)

Python server:

from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import datetime


def get_ms_since_start(start=False):
    global start_ms
    cur_time = datetime.datetime.now()
    # I made sure to stay within hour boundaries while making requests
    ms = cur_time.minute*60000 + cur_time.second*1000 + int(cur_time.microsecond/1000)
    if start:
        start_ms = ms
        return 0
    else:
        return ms - start_ms

class MyServer(BaseHTTPRequestHandler, object):
    def do_GET(self):
        print "Start get method at %d ms" % get_ms_since_start(True)
        field_data = self.path
        self.send_response(200)
        self.end_headers()
        self.wfile.write(str(field_data))
        print "Sent response at %d ms" % get_ms_since_start()
        return

    def do_POST(self):
        print "Start post method at %d ms" % get_ms_since_start(True)
        length = int(self.headers.getheader('content-length'))
        print "Length to read is %d at %d ms" % (length, get_ms_since_start())
        field_data = self.rfile.read(length)
        print "Reading rfile completed at %d ms" % get_ms_since_start()
        self.send_response(200)
        self.end_headers()
        self.wfile.write(str(field_data))
        print "Sent response at %d ms" % get_ms_since_start()
        return


if __name__ == '__main__':
    server = HTTPServer(('0.0.0.0', 8082), MyServer)
    print 'Starting server, use <Ctrl-C> to stop'
    server.serve_forever()

Get request with python server is very fast

time curl -i http://0.0.0.0:8082\?one\=1

prints

HTTP/1.0 200 OK
Server: BaseHTTP/0.3 Python/2.7.6
Date: Sun, 18 Sep 2016 07:13:47 GMT

/?one=1curl http://0.0.0.0:8082\?one\=1  0.00s user 0.00s system 45% cpu 0.012 total

and on the server side:

Start get method at 0 ms
127.0.0.1 - - [18/Sep/2016 00:26:30] "GET /?one=1 HTTP/1.1" 200 -
Sent response at 0 ms

Instantaneous!

Post request when sending form to python server is very slow

time curl http://0.0.0.0:8082 -F one=1

prints

--------------------------2b10061ae9d79733
Content-Disposition: form-data; name="one"

1
--------------------------2b10061ae9d79733--
curl http://0.0.0.0:8082 -F one=1  0.00s user 0.00s system 0% cpu 1.015 total

and on the server side:

Start post method at 0 ms
Length to read is 139 at 0 ms
Reading rfile completed at 1002 ms
127.0.0.1 - - [18/Sep/2016 00:27:16] "POST / HTTP/1.1" 200 -
Sent response at 1002 ms

Specifically, self.rfile.read(length) is taking 1 second for very little form data

Post request when sending data (not form) to python server is very fast

time curl -i http://0.0.0.0:8082 -d one=1

prints

HTTP/1.0 200 OK
Server: BaseHTTP/0.3 Python/2.7.6
Date: Sun, 18 Sep 2016 09:09:25 GMT

one=1curl -i http://0.0.0.0:8082 -d one=1  0.00s user 0.00s system 32% cpu 0.022 total

and on the server side:

Start post method at 0
Length to read is 5 at 0
Reading rfile completed at 0
127.0.0.1 - - [18/Sep/2016 02:10:18] "POST / HTTP/1.1" 200 -
Sent response at 0

node.js server:

var http = require('http');
var qs = require('querystring');

http.createServer(function(req, res) {
    if (req.method == 'POST') {
        whole = ''
        req.on('data', function(chunk) {
            whole += chunk.toString()
        })

        req.on('end', function() {
            console.log(whole)
            res.writeHead(200, 'OK', {'Content-Type': 'text/html'})
            res.end('Data received.')
        })
    }
}).listen(8082)

Post request when sending form to node.js server is very fast

time curl -i http://0.0.0.0:8082 -F one=1

prints:

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: text/html
Date: Sun, 18 Sep 2016 10:31:38 GMT
Connection: keep-alive
Transfer-Encoding: chunked

Data received.curl -i http://0.0.0.0:8082 -F one=1  0.00s user 0.00s system 42% cpu 0.013 total
1
  • If you specify a small size, like 5 to read() on self.rfile (the second argument), does it go faster?
    – datasedai
    Commented Sep 18, 2016 at 12:12

1 Answer 1

6

I think this is the answer to your problem: libcurl delays for 1 second before uploading data, command-line curl does not

libcurl is sending the Expect 100-Continue header, and waiting 1 second for a response before sending the form data (in the case of the -F command).

In the case of -d, it does not send the 100-Continue header, for whatever reason.

1
  • 1
    Using self.send_response(100); self.end_headers() at the start of my do_POST method solved the issue. Many thanks!
    – user34812
    Commented Sep 18, 2016 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.