22

I've thousands of png files which I like to make smaller with pngcrush. I've a simple find .. -exec job, but it's sequential. My machine has quite some resources and I'd make this in parallel.

The operation to be performed on every png is:

pngcrush input output && mv output input

Ideally I can specify the maximum number of parallel operations.

Is there a way to do this with bash and/or other shell helpers? I'm Ubuntu or Debian.

0

3 Answers 3

36

You can use xargs to run multiple processes in parallel:

find /path -print0 | xargs -0 -n 1 -P <nr_procs> sh -c 'pngcrush $1 temp.$$ && mv temp.$$ $1' sh

xargs will read the list of files produced by find (separated by 0 characters (-0)) and run the provided command (sh -c '...' sh) with one parameter at a time (-n 1). xargs will run <nr_procs> (-P <nr_procs>) in parallel.

4
  • $1 does not get populated, I also tried it with a minimal example without luck. xargs is 4.4.0, any idea?
    – mark
    Sep 28, 2010 at 13:15
  • I forgot to specify the value for $0. Should be fixed now.
    – Bart Sas
    Sep 28, 2010 at 13:37
  • @BartSas What does the last 'sh' mean in sh -c '...' sh? Thanks!
    – Dagang
    Aug 29, 2012 at 3:17
  • @Todd It is the value of $0. You can pass any value you want.
    – Bart Sas
    Aug 30, 2012 at 11:54
11

You can use custom find/xargs solutions (see Bart Sas' answer), but when things become more complex you have -at least- two powerful options:

  1. parallel (from package moreutils)
  2. GNU parallel
7
  • 2
    Deb and RPM packages: build.opensuse.org/package/… Mar 26, 2012 at 22:06
  • 4
    I highly recomment GNU parallel over the moreutils version. It is much more flexible. Mar 26, 2012 at 22:07
  • 3
    find /path -print0 | parallel -0 pngcrush {} {.}.temp '&&' mv {.}.temp {}
    – Ole Tange
    Jul 21, 2014 at 6:01
  • 1
    @OleTange that's a great snippet and worth of its own answer so it gets the visibility it deserves! Aug 12, 2014 at 22:52
  • @DrewNoakes Apparently it was not worth its own answer: -1 vote. Oh well.
    – Ole Tange
    Aug 15, 2014 at 23:52
2

With GNU Parallel http://www.gnu.org/software/parallel/ it can be done like:

find /path -print0 | parallel -0 pngcrush {} {.}.temp '&&' mv {.}.temp {} 

Learn more:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.