2

I need the 2nd field of every line in multiple files. 'cut' doesn't work because some lines have leading spaces.

perl -anle 'print $F[1]' *manyfiles* > result 

does work, but is slow.

Is there a significantly faster way to do this?

8 Answers 8

4

I am guessing that the autosplit mode is slow because the entire array obtained by splitting a line has to be stored in memory. This is particularly relevant if your files have long lines. How about this:

perl -ne 'print $1, "\n" if m/^\s*\S+\s+(\S+)/'

Here we do not process the part of the line beyond the second word. You can also test the performance when you use index and substr in place of regular expressions.

0
3
awk '{print $2}' files ... > result

may be faster.

2

could you not remove the leading spaces with sed before using your cut-script?

for example: sed -e 's/^[ \t]*//' would produce a stream with the file without leading spaces. Just pipe this into your script.

1
  • Especially where you're running scripts on a fast multicore machine, you'll see benefits from splitting up your processing into a pipeline of processes. Just make sure you're actually splitting the task up and not duplicating the effort. For example, if Perl already expends time removing leading whitespace, also running sed will only use more CPU time. Even where sed and Perl are running on discrete cores, the pipeline overhead may be noticeable.
    – ctt
    Apr 7, 2012 at 17:32
1
sed -rn 's/\s*[^\s]+\s+([^\s]+).*/\1/p' file1 file2 > parsed_text

Should be faster.

Or you can use this for building a list of files:

find /path/to/files/ -type f -iname "*" -print0 | xargs -0 -I {} sed … 

(‘iname’ just for example of mask, will be more faster if you will not use it)

1

Parallel::ForkManager might help, especially if you do not need the output to be grouped by source file. However, increasing the number of processes simutaneously accessing the disk may also cause a slowdown, but it's worth a shot.

The following example is adopted from the Parallel::ForkManager man page (and obvious errors present in the former version corrected):

#!/usr/bin/env perl

use strict; use warnings;

use Parallel::ForkManager;

my ($maxproc) = @ARGV;
my @files = ('01' .. '10');

my $pm = Parallel::ForkManager->new($maxproc);

for my $file (@files) {
    my $pid = $pm->start and next;
    my $ret = open my $h, '<', $file;

    unless ($ret) {
        warn "Cannot open '$file': $!";
        $pm->finish;
    }

    while (my $line = <$h>) {
        next unless $line =~ /^\s*\S+\s+(\S+)/;
        print "$1\n";
    }

    $pm->finish;
}

$pm->wait_all_children;

I ran the script above with 10 files with 1_000_000 lines each. In each file, 20% of lines had some leading whitespace. See Can Parallel::ForkManager speed up a seemingly IO bound task? for details.

# sync
# echo 3 > /proc/sys/vm/drop_caches 
$ /usr/bin/time -f '%Uuser %Ssystem %Eelapsed %PCPU' ./process.pl 1 > output
24.44user 0.93system 0:29.08elapsed 87%CPU

$ rm output
# sync
# echo 3 > /proc/sys/vm/drop_caches 
$ /usr/bin/time -f '%Uuser %Ssystem %Eelapsed %PCPU' ./process.pl 2 > output
24.95user 0.91system 0:18.31elapsed 141%CPU

$ rm output
# sync
# echo 3 > /proc/sys/vm/drop_caches 
$ /usr/bin/time -f '%Uuser %Ssystem %Eelapsed %PCPU' ./process.pl 4 > output
24.70user 0.88system 0:17.45elapsed 146%CPU

$ rm output 
# sync
# echo 3 > /proc/sys/vm/drop_caches 
$ /usr/bin/time -f '%Uuser %Ssystem %Eelapsed %PCPU' ./process.pl 1 > output
25.31user 0.95system 0:29.72elapsed 88%CPU

So it seems to me there is some gain from utilizing all your cores.

I did not try any of the other suggestions given to see if using Perl+Parallel::ForkManager was better then any of them.

One obvious drawback of this method is that it will interleave lines from the source files. This may or may not matter in your particular situation.

3
  • I wonder how much of this problem is IO bound (which means that forking might not help that much). Apr 11, 2012 at 11:22
  • @briandfoy I did a test on my aging dual core laptop. On Windows, a corrected version of the script ran in about 60% of the time with two processes compared to a single one given 10 files with 1,000,000 lines each (where about 20% of the lines in each file had leading spaces). I'll boot into Linux and try it there as well. The difference seemed to be in CPU utilization with task manager showing both cores utilized at 100% with two processes. I'll post a corrected script and results after I try this on Linux. Apr 11, 2012 at 13:00
  • @briandfoy It seems like there indeed is some benefit to utilizing all available cores. See my updated answer. Apr 11, 2012 at 14:46
0

Perl:

perl -ne 'print "$1\n" if m/\s*\S+\s+(\S+)/' manyfiles >result

non-Perl:

awk '{print $2}' manyfiles >result
0

If the following isn't good enough,

perl -nE'say /^\s*\S+\s+(\S+)/' *

I'd try

perl -ple's/^\s+//' * | cut

If this isn't a one-time thing and speed really matters, you could write a small trimming tool in C to replace perl in the above.

0

I wanted to put some numbers to these solutions. The aaqp.txt file is 130 MB with 1,000,000 lines with an average of 7 fields be line. I actually generated 50+GB of sample data for this, but I was too impatient for waiting for any of these to finish.

$ time perl -anle 'print $F[1]' aaqg.txt > result

real    0m18.526s
user    0m18.368s
sys     0m0.089s

$ time awk '{print $2}' aaqg.txt  > result

real    0m4.051s
user    0m3.592s
sys     0m0.091s

$ time perl -nE 'say $1 if m/\s*\S+\s+(\S+)/' aaqg.txt > result

real    0m2.009s
user    0m1.901s
sys     0m0.066s

$ time perl -nE'say /^\s*\S+\s+(\S+)/' aaqg.txt > result

real    0m2.069s
user    0m1.813s
sys     0m0.069s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.