64

I'm trying to get the 'hello world' of streaming responses working for Django (1.2). I figured out how to use a generator and the yield function. But the response still not streaming. I suspect there's a middleware that's mucking with it -- maybe ETAG calculator? But I'm not sure how to disable it. Can somebody please help?

Here's the "hello world" of streaming that I have so far:

def stream_response(request):
    resp = HttpResponse( stream_response_generator())
    return resp

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)
1

2 Answers 2

52

You can disable the ETAG middleware using the condition decorator. That will get your response to stream back over HTTP. You can confirm this with a command-line tool like curl. But it probably won't be enough to get your browser to show the response as it streams. To encourage the browser to show the response as it streams, you can push a bunch of whitespace down the pipe to force its buffers to fill. Example follows:

from django.views.decorators.http import condition

@condition(etag_func=None)
def stream_response(request):
    resp = HttpResponse( stream_response_generator(), content_type='text/html')
    return resp

def stream_response_generator():
    yield "<html><body>\n"
    for x in range(1,11):
        yield "<div>%s</div>\n" % x
        yield " " * 1024  # Encourage browser to render incrementally
        time.sleep(1)
    yield "</body></html>\n"
5
  • 5
    In my tests the Django GZipMiddleware can prevent this from working.
    – Xealot
    Commented Dec 21, 2010 at 5:55
  • 1
    Yeah, I expect a lot of middlewares are likely to mess with it, so if it's not working, try disabling all your middlewares and re-enable them incrementally. GZip requires having the entire response before it compresses it, so it won't let you stream.
    – Leopd
    Commented Jan 3, 2011 at 1:25
  • 1
    @Xealot: I'm having a similar problem with GzipMiddleware. Filed a bug because that middleware does not support generators (it actually clears the generator accidentally): Django ticket #15066
    – AndiDog
    Commented Jan 12, 2011 at 23:54
  • 1
    I disabled the GZipMiddleware on this call by inserting a fake header into the StreamingHttpResponse: response['Content-Encoding'] = 'identity'. This is probably invalid HTTP in eighteen different ways, but it was just an internal endpoint in my case, so this was good enough for me. Commented Oct 19, 2015 at 21:57
  • Maybe a better way would be to subclass it and customize it to ignore all responses with a header X-Streaming or something like than on.
    – skywalker
    Commented Jan 11, 2016 at 15:31
45

A lot of the django middleware will prevent you from streaming content. Much of this middleware needs to be enabled if you want to use the django admin app, so this can be an annoyance. Luckily this has been resolved in the django 1.5 release. You can use the StreamingHttpResponse to indicate that you want to stream results back and all the middleware that ships with django is aware of this and acts accordingly to not buffer your content output but send it straight down the line. Your code would then look like the following to use the new StreamingHttpResponse object.

def stream_response(request):
    return StreamingHttpResponse(stream_response_generator())

def stream_response_generator():
    for x in range(1,11):
        yield "%s\n" % x  # Returns a chunk of the response to the browser
        time.sleep(1)

Note on Apache

I tested the above on Apache 2.2 with Ubuntu 13.04. The apache module mod_deflate which was enabled by default in the setup I tested will buffer the content you are trying to stream until it reaches a certain block size then it will gzip the content and send it to the browser. This will prevent the above example from working as desired. One way to avoid this is to disable mod_deflate by putting the following line in your apache configuration:

SetEnvIf Request_URI ^/mysite no-gzip=1

This is discussed more in the How to disable mod_deflate in apache2? question.

5
  • It works! but how am I supposed to render it to the template real-time ? Currently I am using Javascript with an ajax call.. but it posts the output.. only when it's finished processing.. So it's not Streaming when it comes to rendering it to the browser. Any help would be appreciated.. Thank you :) Commented Nov 5, 2013 at 20:56
  • @FirstBlood With ajax you can get callbacks as it is streaming and not just when the download is complete. Check out this answer stackoverflow.com/a/4488132/1699750 for showing progress on an Ajax call. Commented Nov 6, 2013 at 12:02
  • Thanks.. I want to achieve Ping output status.. to be printed real-time on the browser.. So for ex: ping -c 3 www.google.com .. how the command prompt gives the output.. is it possible in Django, to spill out the output to the browser as I read via subprocess? I tried the StreamingHttpResponse, but it doesn't seem to stream real-time.. I even tried setting headers: Keep-Alive.. but no effect :( Commented Nov 6, 2013 at 20:30
  • your webserver (eg Apache) will likely also buffer the output, if the process is not very long running or doesn't send much output this may have the effect that the webserver buffers it all and then delivers it in one lump at the end
    – Anentropic
    Commented Mar 29, 2014 at 9:38
  • @Anentropic That is true. I have updated the answer with a note about Apache Commented Mar 29, 2014 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.