113

How can I randomize the lines in a file using standard tools on Red Hat Linux?

I don't have the shuf command, so I am looking for something like a perl or awk one-liner that accomplishes the same task.

3

11 Answers 11

226

Um, lets not forget

sort --random-sort
9
  • 1
    Well, I'm using gnu-coreutils 7.1 (standard gentoo install), which has sort with this option, not sure when it appeared, or if it's in other implementations.
    – Jim T
    May 21, 2009 at 11:46
  • 1
    The feature was committed on 10th December 2005, the release following that was 5.94, so I'm guessing it's been available since that version.
    – Jim T
    May 21, 2009 at 11:58
  • 41
    On OS X you can install gnu coreutils with homebrew: brew install coreutils All the utils are prefixed with a g so: gsort --random-sort or gshuf will work as expected
    – mike
    Aug 21, 2013 at 4:14
  • 3
    +1 @mike. I use Macports and I also had gsort and gshuf installed when I did port install coreutils Sep 4, 2013 at 3:06
  • 10
    This solution is only good if your lines do not have repetitions. If they do, all instances of that line will appear next to each other. Consider using shuf instead (on linux).
    – Ali J
    May 7, 2014 at 20:16
131

shuf is the best way.

sort -R is painfully slow. I just tried to sort 5GB file. I gave up after 2.5 hours. Then shuf sorted it in a minute.

7
  • This is great. It appears to be in GNU coreutils.
    – ariddell
    Mar 27, 2013 at 13:06
  • 4
    I suspect the reason sort -R is slow is that computes a hash for each line. From the docs: "Sort by hashing the input keys and then sorting the hash values."
    – Joe Flynn
    Jun 13, 2013 at 18:40
  • 16
    beware, shuf loads everything in memory.
    – jfs
    Oct 27, 2014 at 7:54
  • 1
    @benroth: From what I can tell, with really large input counts increasing the memory can help somewhat, but it's still slow overall. In my tests, sorting a 1-million-line input file created with seq -f 'line %.0f' 1000000 took the same, long time to process (much, much longer than with shuf), no matter how much memory I allocated.
    – mklement0
    May 8, 2015 at 23:00
  • 1
    @mklement0, you are right! I just tried it with a much bigger file than what I had before, and the hashing seems to be the bottleneck indeed.
    – benroth
    May 11, 2015 at 18:41
66

And a Perl one-liner you get!

perl -MList::Util -e 'print List::Util::shuffle <>'

It uses a module, but the module is part of the Perl code distribution. If that's not good enough, you may consider rolling your own.

I tried using this with the -i flag ("edit-in-place") to have it edit the file. The documentation suggests it should work, but it doesn't. It still displays the shuffled file to stdout, but this time it deletes the original. I suggest you don't use it.

Consider a shell script:

#!/bin/sh

if [[ $# -eq 0 ]]
then
  echo "Usage: $0 [file ...]"
  exit 1
fi

for i in "$@"
do
  perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new
  if [[ `wc -c $i` -eq `wc -c $i.new` ]]
  then
    mv $i.new $i
  else
    echo "Error for file $i!"
  fi
done

Untested, but hopefully works.

3
  • To backup the original file, you can suffix an extension to the -i flag [perldoc.perl.org/perlrun.html] May 25, 2009 at 8:11
  • I'm usually a Perl fan, but came across this ruby example which has the benefit of being shorter: ruby -e 'puts STDIN.readlines.shuffle'. It would need testing on big inputs to see if the speed is comparable. (also works on OS X)
    – mivk
    May 17, 2015 at 21:48
  • per comment below, shuf loads everything into memory, so it doesn't work with a truly huge file (mine is ~300GB tsv). This perl script failed on mine too, but with no error except Killed. Any idea if the perl solution is loading everything into memory too, or is there some other problem I'm encountering?
    – seth127
    Jan 17, 2018 at 15:05
24
cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7-

Read the file, prepend every line with a random number, sort the file on those random prefixes, cut the prefixes afterwards. One-liner which should work in any semi-modern shell.

EDIT: incorporated Richard Hansen's remarks.

11
  • 1
    This works, and is a creative solution, but will delete leading whitespace on lines.
    – Chris Lutz
    May 20, 2009 at 5:39
  • @Chris changing the last cut to |sed 's/^[^\t]*\t//' should fix that
    – bdonlan
    May 20, 2009 at 5:43
  • Kudos to the simplicity of the approach! May 20, 2009 at 6:22
  • 3
    +1 for POSIX conformance (except for $RANDOM), but -1 for butchering the data. Replacing while read f with while IFS= read -r f will prevent read from removing leading and trailing whitespace (see this answer) and prevent processing of backslashes. Using a fixed-length random string will prevent cut from deleting leading whitespace. Result: cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7- Mar 29, 2013 at 19:17
  • 3
    @Richard Hansen: Thanks, these suggested changes are obviously appropriate, I've edited my post. Apr 8, 2013 at 22:49
10

A one-liner for python:

python -c "import random, sys; lines = open(sys.argv[1]).readlines(); random.shuffle(lines); print ''.join(lines)," myFile

And for printing just a single random line:

python -c "import random, sys; print random.choice(open(sys.argv[1]).readlines())," myFile

But see this post for the drawbacks of python's random.shuffle(). It won't work well with many (more than 2080) elements.

0
5

Related to Jim's answer:

My ~/.bashrc contains the following:

unsort ()
{
    LC_ALL=C sort -R "$@"
}

With GNU coreutils's sort, -R = --random-sort, which generates a random hash of each line and sorts by it. The randomized hash wouldn't actually be used in some locales in some older (buggy) versions, causing it to return normal sorted output, which is why I set LC_ALL=C.


Related to Chris's answer:

perl -MList::Util=shuffle -e'print shuffle<>'

is a slightly shorter one-liner. (-Mmodule=a,b,c is shorthand for -e 'use module qw(a b c);'.)

The reason giving it a simple -i doesn't work for shuffling in-place is because Perl expects that the print happens in the same loop the file is being read, and print shuffle <> doesn't output until after all input files have been read and closed.

As a shorter workaround,

perl -MList::Util=shuffle -i -ne'BEGIN{undef$/}print shuffle split/^/m'

will shuffle files in-place. (-n means "wrap the code in a while (<>) {...} loop; BEGIN{undef$/} makes Perl operate on files-at-a-time instead of lines-at-a-time, and split/^/m is needed because $_=<> has been implicitly done with an entire file instead of lines.)

2
  • Reiterating that sort -R doesn't exist on OS X, but +1 for some great Perl answers, and a great answer in general.
    – Chris Lutz
    May 20, 2009 at 16:40
  • You could install GNU coreutils on OS X, but (as I've done in the past) you have to be careful not to break the built-in tools... That being said, OP is on Redhat Linux, which definitely has GNU coreutils standard.
    – ephemient
    May 20, 2009 at 16:49
3

When I install coreutils with homebrew

brew install coreutils

shuf becomes available as n.

2
  • brew prefixed all commands with g so shufbecame gshuffor me.
    – Jörn
    Jun 13, 2016 at 21:16
  • ^ Is that because they're non-POSIX, or am I just totally off?
    – Dave Liu
    Jul 19, 2016 at 22:26
1

Mac OS X with DarwinPorts:

sudo port install unsort
cat $file | unsort | ...
1

FreeBSD has its own random utility:

cat $file | random | ...

It's in /usr/games/random, so if you have not installed games, you are out of luck.

You could consider installing ports like textproc/rand or textproc/msort. These might well be available on Linux and/or Mac OS X, if portability is a concern.

-1

On OSX, grabbing latest from http://ftp.gnu.org/gnu/coreutils/ and something like

./configure make sudo make install

...should give you /usr/local/bin/sort --random-sort

without messing up /usr/bin/sort

2
  • this didn't work for me on OSX (10.7). I got "configure: error: C compiler cannot create executables". Jul 24, 2011 at 17:43
  • @dolan Check your permissions?
    – Benubird
    May 24, 2013 at 8:22
-1

Or get it from MacPorts:

$ sudo port install coreutils

and/or

$ /opt/local//libexec/gnubin/sort --random-sort

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.