0

First of all sorry for my poor English. Please let me explain my scenario first, I have data stored in Memcached as a group of gzip data. I call each a chunk of data. Each is loaded from the database, compressed and stored in Memcached in advance. This is done as a preparation process before the service can be available. (To reduce load on the database)

My server code then simply fetches the gzip data from Memcached and then returns the gzip data immediately.

From users's perspective, they can request any number of chunks at the same time. For example, assume that there are 10 chunks on Memcached, a user may request a single chunk, which in this case I can send back a gzip file as a response without any problem. But when a user requests a number of chunks at the same time, I simply don't know how to deal with it. Getting a number of chunks and compressing them as a single GZIP file is not a solution. I simple want to send requested gzip files simultaneously as a single request.

My first thought is that I will load data from the database, and then store them in Memcached without compressing them. When a request comes in, I will get a stored data in Memcached and then compress it and send it back. But doing this is surely a waste of cpu cycle since the compression must be done every time the response is created. So I want a way not to compress them, just send them back immediately and the browser still understand what I send.

If my question puzzles you, please let me know I will try to elaborate it. Because I really need helps.

1

2 Answers 2

1

You'll need to come up with some sort of container format for the data, basically. That could be as simple as sending a single response consisting of:

  • Length of data for chunk 1
  • Data for chunk 1
  • Length of data for chunk 2
  • Data for chunk 2

(where each chunk is already gzipped, of course).

Alternatively you could create a zip file with no compression to contain multiple "files" each of which is a gzipped chunk. It really depends on what your client is going to do with the result. What is the client here, exactly?

6
  • First of all, Thank Jon for a quick reply. The client is a normal web browser. As for the first approach you proposed, I am not sure if the
    – woraphol.j
    Sep 19, 2011 at 6:41
  • @s4510488: If the client is a normal web browser, then you need to think about how you want the result to appear to the user. It sounds like a zip file may be the best bet.
    – Jon Skeet
    Sep 19, 2011 at 6:43
  • The first comment is unintentionally not complete but it seems you understand what I am asking. As I understand, when the browser receives the response, whose MIME type is gzip, it just unzip and parse what is in the zip file and present them to the user, doesn't it? I am not sure whether the browser can understand gzip in gzip.
    – woraphol.j
    Sep 19, 2011 at 6:57
  • @s4510488: No, the browser won't understand that - basically I believe the built-in gzip compression is really only applicable to a single piece of gzipped content. If there are multiple chunks and you wish the client to see them uncompressed, I think you'll need to build a new zip file each time rather than using pre-compressed data. But compression can be pretty quick...
    – Jon Skeet
    Sep 19, 2011 at 7:14
  • Although compression can be quick, but it still suffers if the data is large. I expect each chunk of my data is around 8k. A user can request as many as 100 chunks which can result in the data that is as large as 800k. So the server will suffer from this. Imagine that If there were a thousand of requests within seconds, the server would spend most of the time compressing the same data over and over again. So If possible, I want to avoid this. But the way, thank for you answers, Jon :)
    – woraphol.j
    Sep 19, 2011 at 8:53
0

HTTP itself supports gzip compression. I suggest you use it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.