2

I have 2 files, a small one and a big one. The small file is a subset of the big one.

For instance:

Small file:

solar:1000
alexey:2000

Big File:

andrey:1001
solar:1000
alexander:1003
alexey:2000

I want to delete all the lines from Big.txt which are also present in Small.txt. In other words, I want to delete the lines in Big file which are common to the small File.

So, I wrote a Perl Script as shown below:

#! /usr/bin/perl

use strict;
use warnings;

my ($small, $big, $output) = @ARGV;

open(BIG, "<$big") || die("Couldn't read from the file: $big\n");
my @contents = <BIG>;
close (BIG);

open(SMALL, "<$small") || die ("Couldn't read from the file: $small\n");

while(<SMALL>)
{
    chomp $_;
    @contents = grep !/^\Q$_/, @contents;
}

close(SMALL);

open(OUTPUT, ">>$output") || die ("Couldn't open the file: $output\n");

print OUTPUT @contents;
close(OUTPUT);

However, this Perl Script does not delete the lines in Big.txt which are common to Small.txt

In this script, I first open the big file stream and copy the entire contents into the array, @contents. Then, I iterate over each entry in the small file and check for its presence in the bigger file. I filter the line from Big File and save it back into the array.

I am not sure why this script does not work? Thanks

4
  • 3
    If you want a non perl solution: comm -1 -3 file1 file2 > file2
    – Vikas
    Jun 4, 2012 at 12:12
  • @TLP - is that limited to GNU grep? And Also, shouldn't it include -v too?
    – DVK
    Jun 4, 2012 at 12:34
  • @DVK You would have to ask someone else about that. Yes, -v to invert and probably -x to match entire line as ysth says too.
    – TLP
    Jun 4, 2012 at 12:41
  • grep can do this, but needs more switches: grep -F -v -x -f smallfile bigfile
    – ysth
    Jun 4, 2012 at 12:42

3 Answers 3

4

Your script does NOT work because grep uses $_ and takes over (for the duration of grep) the old value of your $_ from the loop (e.g. the variable $_ you use in the regex is NOT the variable used for storing the loop value in the while block - they are named the same, but have different scopes).

Use a named variable instead (as a rule, NEVER use $_ for any code longer than 1 line, precisely to avoid this type of bug):

while (my $line=<SMALL>) {
    chomp $line;
    @contents = grep !/^\Q$line/, @contents;
}

However, as Oleg pointed out, a more efficient solution is to read small file's lines into a hash and then process the big file ONCE, checking hash contents (I also improved the style a bit - feel free to study and use in the future, using lexical filehandle variables, 3-arg form of open and IO error printing via $!):

#! /usr/bin/perl

use strict;
use warnings;

my ($small, $big, $output) = @ARGV;

use File::Slurp;
my @small = read_file($small);
my %small = map { ($_ => 1) } @small;

open(my $big, "<", $big) or die "Can not read $big: Error: $!\n";
open(my $output, ">", $output) or die "Can not write to $output: Error: $!\n";

while(my $line=<$big>) {
    chomp $line;
    next if $small{$line}; # Skip common
    print $output "$line\n";
}

close($big);
close($output);
5
  • The grep solution also needs an end anchor or more may be filtered out than desired: /^\Q$line\E\n/
    – ysth
    Jun 4, 2012 at 12:35
  • @ysth - while not the main problem with the script, IMHO your comment is worth a separate answer
    – DVK
    Jun 4, 2012 at 12:38
  • ..NEVER use $_ for any code longer than 1 line -- sounds excessive. I would say as a general rule, do not use $_ for more lines than your (boss's?) proficiency and memory allows. The different $_ are not overwritten, they are just overshadowed, similar to for local $_ (@foo) ..
    – TLP
    Jun 4, 2012 at 12:55
  • @TLP - you're correct, bad choice of words. It's "shifted out of scope" :) And personally, when I'm reading code in a production emergency at 2am, I do NOT want to depend on proficiency or memory. I want it to be as crystal clear and obvious as possible. Meaning, self-documenting identifiers to the max.
    – DVK
    Jun 4, 2012 at 13:01
  • @DVK Readability is good, I just think that one line as a rule of thumb is a bit extreme. But you are right in the sense that for the most extreme situations, extreme measures are prudent.
    – TLP
    Jun 4, 2012 at 13:22
3

It doesn't work for several reasons. First, lines in @content still have their newlines in. And second, when you grep, $_ in !/^\Q$_/ is set not to the last line from small file, but for each element of @contents array, effectively making it: for each element in list return everything except this element, leaving you with empty list at the end.

This isn't really the good way to do it - you're reading big file and then trying to reprocess it several times. First, read a small file and put every line in hash. Then read big file inside while(<>) loop, so you won't waste your memory reading it entirely. On each line, check if key exists in previously populated hash and if it does - go to next iteration, otherwise print the line.

2
  • This is perfect. But his question was: why does his script not work? Despite your way being obviously much better, his should work too, no?
    – Konerak
    Jun 4, 2012 at 12:14
  • Thanks for your help. I will try to implement it the way you suggested by using hashes as well. I knew there was a better way to implement, I was trying to do it quickly, but I will learn the way you suggested too.
    – Neon Flash
    Jun 4, 2012 at 12:31
1

Here is a small and efficient solution to your problem:

#!/usr/bin/perl

use strict;
use warnings;

my ($small, $big, $output) = @ARGV;

my %diffx;

open my $bfh, "<", $big or die "Couldn't read from the file $big: $!\n";
# load big file's contents
my @big = <$bfh>;
chomp @big;
# build a lookup table, a structured table for big file
@diffx{@big} = ();
close $bfh or die "$!\n";

open my $sfh, "<", $small or die "Couldn't read from the file $small: $!\n";
my @small = <$sfh>;
chomp @small;
# delete the elements that exist in small file from the lookup table
delete @diffx{@small};
close $sfh;

# print join "\n", keys %diffx;

open my $ofh, ">", $output or die "Couldn't open the file $output for writing: $!\n";
# what is left is unique lines from big file
print $ofh join "\n", keys %diffx;  
close $ofh;

__END__

P.S. I learned this trick and many others from Perl Cookbook, 2nd Edition. Thanks

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.