39

Problem

I'm creating multiple charts that are then sent to the server to create a PDF. This data can get large.

Question

What is the best way to compress and send the image data to the server

Background

The charts I'm creating are fairly complex, to save myself the headache all of this is converted to a canvas where the base64 data uri is generated. Currently the data uri(s) are posted to the server to handle the processing. Posted info can get fairly large at around 400-600kb each for a small chart and 12mb for the largest chart.

The charts are org charts that can be manipulated/reordered.

Is there a better method of compressing these strings before sending it back up to the server?

Research

Some things I've check out:

https://github.com/brunobar79/J-I-C/blob/master/src/JIC.js (does not look like an option just resizes)

http://pastebin.com/MhJZBsqW (looks interesting)

But references external libraries that I cannot find: C_H_Image Lib_MinifyJpeg

https://code.google.com/p/jslzjb/source/browse/trunk/Iuppiter.js?r=2 (looks like this could work but relies on decompression on server side)

7
  • 3
    I dont think 400kb is that big.anyway these charts should be created on the server at first place.A chart is a representation of data.Just send the data to the server and make the server render the charts. You dont want to do any compression client side.
    – mpm
    Mar 13, 2014 at 22:11
  • 1
    @mpm thanks for the reply, the reason I want to persist the same chart is because the charts are organization charts that can be manipulated (drag and dropped). I have not found a way to just generate the charts server side in a similar way as the front end. The data being posted can get up to 12MB, which should be okay but I would love to cut down the size of the image data. Mar 14, 2014 at 13:38
  • What do use to generate the charts? Maybe you can send the data to the server and generate the charts with the same methods you use on the client but with a headless browser.
    – Koen.
    Mar 14, 2014 at 15:14
  • @Koen the charts are created using divs + svg client side. I think a headless browser would introduce more complex issues in the system so I stayed away from going that direction. Mar 14, 2014 at 20:23
  • Split the base64 string and send the parts as chunks ( can give your visitors and upload status easy then too ) Mar 14, 2014 at 20:30

3 Answers 3

54
+100

Maybe string compression is the solution for you. This converts the data to byte arrays.

There are multiple implementations and algorithms around, for instance

  • LZMA-JS A standalone JavaScript implementation of the Lempel-Ziv-Markov chain (LZMA) compression algorithm.

    my_lzma = new LZMA("./lzma_worker.js");
    my_lzma.compress("This is my compression test.", 1, function on_compress_complete(result) {
        console.log("Compressed: " + result);
        my_lzma.decompress(result, function on_decompress_complete(result) {
            console.log("Decompressed: " + result);
        }, function on_decompress_progress_update(percent) {
            console.log("Decompressing: " + (percent * 100) + "%");
        });
    }, function on_compress_progress_update(percent) {
        console.log("Compressing: " + (percent * 100) + "%");
    });
    
  • lz-string: JavaScript compression, fast!

    var str = "This is my compression test.";
    console.log("Size of sample is: " + str.length);
    var compressed = LZString.compress(str);
    console.log("Size of compressed sample is: " + compressed.length);
    str = LZString.decompress(compressed);
    console.log("Sample is: " + str);
    
2
  • Thanks for the suggestion, I don't think it would work out because the input was truncating the data. Mar 17, 2014 at 21:28
  • I tried lz-string, and it appears to not work for images larger than 100x100
    – Seph Reed
    Jul 23, 2017 at 1:21
13

I needed to generate thumbnails from larger pictures. I decided to solve my version of this problem with the HTML5 Canvas technique. I am using GWT so my code is:

//Scale to size
public static String scaleImage(Image image, int width, int height) {

    Canvas canvasTmp = Canvas.createIfSupported();
    Context2d context = canvasTmp.getContext2d();

    canvasTmp.setCoordinateSpaceWidth(width);
    canvasTmp.setCoordinateSpaceHeight(height);

    ImageElement imageElement = ImageElement.as(image.getElement());

    context.drawImage(imageElement, 0, 0, width, height);

    //Most browsers support an extra option for: toDataURL(mime type, quality) where quality = 0-1 double.
    //This is not available through this java library but might work with elemental version?
    String tempStr = canvasTmp.toDataUrl("image/jpeg");

    return tempStr;
}

If you are using JS you can probably get the idea:

  • Make a canvas the size of the desired output image
  • draw the input image on the canvas in the new size
  • call canvas.toDataURL("mime-type", quality)

You can use any mime-type I think, but for me the jpeg one was the smallest and was comparable to results of my desktop image program.

My GWT would not let me do the quality parameter (and I'm not 100% sure how widely supported it is in which browsers), but that was OK because the resulting images were quite small. If you leave the mime-type blank it defaults to png which in my case was 3-4x larger than jpeg.

5

I finally got this working with fairly large files.

To do it, I used lz-string, as shown above.

There is one issue though, which is that sending UTF-16 data via ajax is deprecated, so.. it doesn't work. The fix is to convert to and from UTF-16 and UTF-8.

Here's the conversion code I'm using. I'm not sure what the standard endian is, btw:

var UTF = {};

UTF.U16to8 = function(convertMe) {
var out = "";
  for(var i = 0; i < convertMe.length; i++) {
      var charCode = convertMe.charCodeAt(i);
      out += String.fromCharCode(~~(charCode/256));
      out += String.fromCharCode(charCode%256);
    }
  return out;
}

UTF.U8to16 = function(convertMe) {
  var out = ""
  for(var i = 0; i < convertMe.length; i += 2) {
    var charCode = convertMe.charCodeAt(i)*256;
    charCode += convertMe.charCodeAt(i+1);
    out += String.fromCharCode(charCode)
  }
  return out;
}

If you're using Node.js, this code works on both sides. If not, you'll have to write your own version of these converters for the server side.

Phew! What a day this was.

3
  • I'm compress then convert to UTF-8 before sending to server. In server I convert to UTF-16 then decompress but I get null
    – Akashii
    Jul 23, 2018 at 3:55
  • UTF-16 is not DEPRECATED, just that it has issues... Plus, even if you use it on both ends, enabling compression on the transfer doesn't really solve the amount of data transferred...
    – KolonUK
    Sep 4, 2019 at 13:00
  • 1
    This is cool but for me it loses most of the compression UTF-16 affords. Mar 23, 2022 at 4:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.