1

We have a first version of a service built that needs to take in image and pdf files and then do lots of processing on each one, and serve several resized variants to the web and mobile clients we have built.

On the processing side we perform:

  • 9 image size variants for web and mobile
  • Tiling (a-la map tiling) of 300dpi image
  • 5 image processing and machine learning/tagging processes

In running a test of the full pipeline, in serial, it took around 18 minutes to process around 120 images.

We are trying to cut down that time dramatically. One thing of course would be to do various processing in parallel, there are only a few dependencies, for instance, we cant process the image processing/machine learning steps until a couple key variants are generated.

From a architecture standpoint, we want to offload all of the processing from our web tier, but also need to serve the images up to the web/mobile clients. We have been investigating things like openmpi, hadoop, zeromq for parallelizing things, and hdfs, gluster, lustre for solving the storage/scale problems.

Looking for any references architectures, or tooling that can be stiched together so we can incrementally improve our setup.

1 Answer 1

0

So which image- and PDF-processing tools were you using for the 120 images?!? (You can think and plan all you want about and around [storage], [hadoop] and [openmpi] -- unless your image and PDF processing tools are fast enough your bottleneck will be found there.)

2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.