How can I randomize the lines in a file using standard tools on Red Hat Linux?
I don't have the shuf
command, so I am looking for something like a perl
or awk
one-liner that accomplishes the same task.
How can I randomize the lines in a file using standard tools on Red Hat Linux?
I don't have the shuf
command, so I am looking for something like a perl
or awk
one-liner that accomplishes the same task.
Um, lets not forget
sort --random-sort
brew install coreutils
All the utils are prefixed with a g so: gsort --random-sort
or gshuf
will work as expected
gsort
and gshuf
installed when I did port install coreutils
Sep 4, 2013 at 3:06
shuf
instead (on linux).
shuf
is the best way.
sort -R
is painfully slow. I just tried to sort 5GB file. I gave up after 2.5 hours. Then shuf
sorted it in a minute.
sort -R
is slow is that computes a hash for each line. From the docs: "Sort by hashing the input keys and then sorting the hash values."
Jun 13, 2013 at 18:40
seq -f 'line %.0f' 1000000
took the same, long time to process (much, much longer than with shuf
), no matter how much memory I allocated.
May 8, 2015 at 23:00
And a Perl one-liner you get!
perl -MList::Util -e 'print List::Util::shuffle <>'
It uses a module, but the module is part of the Perl code distribution. If that's not good enough, you may consider rolling your own.
I tried using this with the -i
flag ("edit-in-place") to have it edit the file. The documentation suggests it should work, but it doesn't. It still displays the shuffled file to stdout, but this time it deletes the original. I suggest you don't use it.
Consider a shell script:
#!/bin/sh
if [[ $# -eq 0 ]]
then
echo "Usage: $0 [file ...]"
exit 1
fi
for i in "$@"
do
perl -MList::Util -e 'print List::Util::shuffle <>' $i > $i.new
if [[ `wc -c $i` -eq `wc -c $i.new` ]]
then
mv $i.new $i
else
echo "Error for file $i!"
fi
done
Untested, but hopefully works.
ruby -e 'puts STDIN.readlines.shuffle'
. It would need testing on big inputs to see if the speed is comparable. (also works on OS X)
shuf
loads everything into memory, so it doesn't work with a truly huge file (mine is ~300GB tsv). This perl script failed on mine too, but with no error except Killed
. Any idea if the perl solution is loading everything into memory too, or is there some other problem I'm encountering?
cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7-
Read the file, prepend every line with a random number, sort the file on those random prefixes, cut the prefixes afterwards. One-liner which should work in any semi-modern shell.
EDIT: incorporated Richard Hansen's remarks.
$RANDOM
), but -1 for butchering the data. Replacing while read f
with while IFS= read -r f
will prevent read
from removing leading and trailing whitespace (see this answer) and prevent processing of backslashes. Using a fixed-length random string will prevent cut
from deleting leading whitespace. Result: cat yourfile.txt | while IFS= read -r f; do printf "%05d %s\n" "$RANDOM" "$f"; done | sort -n | cut -c7-
Mar 29, 2013 at 19:17
A one-liner for python:
python -c "import random, sys; lines = open(sys.argv[1]).readlines(); random.shuffle(lines); print ''.join(lines)," myFile
And for printing just a single random line:
python -c "import random, sys; print random.choice(open(sys.argv[1]).readlines())," myFile
But see this post for the drawbacks of python's random.shuffle()
. It won't work well with many (more than 2080) elements.
Related to Jim's answer:
My ~/.bashrc
contains the following:
unsort ()
{
LC_ALL=C sort -R "$@"
}
With GNU coreutils's sort, -R
= --random-sort
, which generates a random hash of each line and sorts by it. The randomized hash wouldn't actually be used in some locales in some older (buggy) versions, causing it to return normal sorted output, which is why I set LC_ALL=C
.
Related to Chris's answer:
perl -MList::Util=shuffle -e'print shuffle<>'
is a slightly shorter one-liner. (-Mmodule=a,b,c
is shorthand for -e 'use module qw(a b c);'
.)
The reason giving it a simple -i
doesn't work for shuffling in-place is because Perl expects that the print
happens in the same loop the file is being read, and print shuffle <>
doesn't output until after all input files have been read and closed.
As a shorter workaround,
perl -MList::Util=shuffle -i -ne'BEGIN{undef$/}print shuffle split/^/m'
will shuffle files in-place. (-n
means "wrap the code in a while (<>) {...}
loop; BEGIN{undef$/}
makes Perl operate on files-at-a-time instead of lines-at-a-time, and split/^/m
is needed because $_=<>
has been implicitly done with an entire file instead of lines.)
When I install coreutils with homebrew
brew install coreutils
shuf
becomes available as n
.
FreeBSD has its own random utility:
cat $file | random | ...
It's in /usr/games/random, so if you have not installed games, you are out of luck.
You could consider installing ports like textproc/rand or textproc/msort. These might well be available on Linux and/or Mac OS X, if portability is a concern.
On OSX, grabbing latest from http://ftp.gnu.org/gnu/coreutils/ and something like
./configure make sudo make install
...should give you /usr/local/bin/sort --random-sort
without messing up /usr/bin/sort
Or get it from MacPorts:
$ sudo port install coreutils
and/or
$ /opt/local//libexec/gnubin/sort --random-sort