29

I'm sending a huge JSON string (with jsonify in Flask) to my webpage pretty often, so I would like to reduce the data. The easiest option is probably to remove all line breaks and space characters, but just to give you an example of this:

Normal jsonify: 361KB
Removing all line breaks and space characters: 118KB (wow).
Zip the original file: 35KB (double wow).

So I basically wonder if there is an easy way to come close to the 35KB. I couldn't find a solution so far which I could easily implement in python and javascript (to decompress).

Right now, I send around 4-5MB of data every second, which is - you guessed right - a "little bit" too much.

6 Answers 6

33

Old question but I was searching for this and it was the first result on Google. The link to the answer of Leon has a solution not for Flask and also it is old. With Python 3 now we can do all in few lines with the standard libraries (and Flask):

from flask import make_response, json
import gzip

@app.route('/data.json')
def compress():
    very_long_content = [{'a': 1, 'b': 2}, {'c': 3, 'd': 4}]
    content = gzip.compress(json.dumps(very_long_content).encode('utf8'), 5)
    response = make_response(content)
    response.headers['Content-length'] = len(content)
    response.headers['Content-Encoding'] = 'gzip'
    return response

With gzip.compress we have directly a byte string compressed and it is required as input a byte string. Then, as the link from Leon, we make a custom response saying that the content is a gzip so the browser will decompress by itself.

For decoding in Javascript using a JQuery ajax request there isn't any particular difference from a standard request:

$.ajax({
    url: '/data.json',
    dataType: 'json',
    success: function(data) {
        console.log(data);
    }
})

Note that this snippet compress and then send the long content. You should consider the amount of time that it takes to compress the content (especially in this case that we have very long content), so be sure to set an appropriate level of compression that doesn't require more time to compress + send than send the long content as it is.

My use case was that I sent the big content from a slow connection, so I had all the benefits to compress the content before send it.

4
  • 4
    In addition I also added a check that the request can handle gzip in a response: if 'gzip' in request.headers.get('Accept-Encoding','').lower() Commented Jul 12, 2020 at 18:19
  • +1 to checking to make sure that the time for compression doesn't actually increase the load time. I implemented this and it actually increased the load time by 200ms! Commented Jan 9, 2022 at 4:29
  • @redorkulated No reason to wing it when Flask (actually Werkzeug below it) has a proper parser: if 'gzip' in request.accept_encodings. Commented Jul 20 at 16:50
  • @AlexShpilkin thanks for the reply, indeed your right , not sure back then (this is a 4 year old question). But, either way, yes it is there today Commented Jul 21 at 17:16
19

Web requests do support GZip and you could implement it in python.

Here is someone who asked that exact question. How to use Content-Encoding: gzip with Python SimpleHTTPServer

According to the flask-compress repo

The preferred solution is to have a server (like Nginx) automatically compress the static files for you.

But you can do it in flask: https://github.com/colour-science/flask-compress.

If you go the gzip route you will not need to remove line breaks and white space, but if you still want to then according to the flask documentation you can disable pretty print by setting JSONIFY_PRETTYPRINT_REGULAR to false.

3
  • Thank you very much for this great response - from the description it seems to me like flask-compress would be perfect, but since it's so simple to integrate I have no Idea if it is actually working. Is there any way to test how much data is actually send / if the compression works?
    – lakerz
    Commented May 11, 2015 at 11:52
  • If the url is publicly visible you can use something like checkgzipcompression.com otherwise using Chrome you can open developer tools and inspect your network traffic. The response should contain a header entry Content-Encoding which is set to gzip
    – Leon
    Commented May 11, 2015 at 11:58
  • 1
    Yes, I just found the DevTools - another reason why Chrome is awesome. There I could see that a 100KB response was only 16.8KB with the easy flask-compression extension. AWESOME. Thanks a lot.
    – lakerz
    Commented May 11, 2015 at 12:08
12

If you're looking for a library to help serve compressed content via Gzip (or Brotli), try flask-compress. It's very simple; this might be all you need to do:

from flask import Flask
from flask_compress import Compress

app = Flask(__name__)
Compress(app)
3

Inspired by Ripper346's answer, but rather than manually writing it for all routes/views. It is just better to add a snippet(compared to external dependency on a library), which will compress the response before sending it out.

Add the following to your app file, or wherever suited(eg. gzip_compress.py):

import gzip
from io import BytesIO
from flask import request

class GzipCompress:
    def __init__(self, app, compress_level=9, minimum_size=100):
        self.app = app
        self.compress_level = compress_level
        self.minimum_size = minimum_size
        self.app.after_request(self.after_request)

    def after_request(self, response):
        accept_encoding = request.headers.get('Accept-Encoding', '')

        if response.status_code < 200 or \
           response.status_code >= 300 or \
           response.direct_passthrough or \
           len(response.get_data()) < self.minimum_size or \
           'gzip' not in accept_encoding.lower() or \
           'Content-Encoding' in response.headers:
            return response

        gzip_buffer = BytesIO()
        gzip_file = gzip.GzipFile(mode='wb', 
                                  compresslevel=self.compress_level, 
                                  fileobj=gzip_buffer)
        gzip_file.write(response.get_data())
        gzip_file.close()
        response.set_data(gzip_buffer.getvalue())
        response.headers['Content-Encoding'] = 'gzip'
        response.headers['Content-Length'] = len(response.get_data())

        return response

And then in your app:

from flask import Flask
from gzip_compress import GzipCompress

app = Flask(__name__)
GzipCompress(app)

You can set your gzip compression level and the minimum response size for compression to kick in. It includes other checks for gzip support and response codes.

1
  • Cool if you want to compress all the routes. Also, you could think about making a static function on GzipCompress to use it only when you want to compress sometimes. You will have all in one place
    – Ripper346
    Commented Dec 16, 2020 at 10:56
3

Spinoff of suvigyavijay's answer, without using awkward classes, io.BytesIO(), or gzip.GzipFile (he did a lot of heavy lifting though, props).

Note. This applies to all requests, I think.

import flask, gzip

app = flask.Flask(__name__)

@app.after_request
def compress(response):
  accept_encoding = flask.request.headers.get('accept-encoding','').lower()
  if response.status_code<200 or response.status_code>=300 or response.direct_passthrough or 'gzip' not in accept_encoding or 'Content-Encoding' in response.headers:  return response
  content = gzip.compress(response.get_data(), compresslevel=9)  # 0: no compression, 1: fastest, 9: slowest. Default: 9
  response.set_data(content)
  response.headers['content-length']   = len(content)
  response.headers['content-encoding'] = 'gzip'
  return response
1

The selected answer talks about Flask-Compress which is great at compressing. However, you may see that your JSON is still not stripped of new lines and indentation (esp when you're checking the output in debug mode).

Since JSONIFY_PRETTYPRINT_REGULAR is deprecated, to get this to work in the later Flask versions (2.2+), set:

app.json.compact = True

It took me some time to find this setting, so adding this here to avoid having to search for it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.