1

I'm working with huge files. I would like to know, in your opinion, what is the best way to handle huge files when you would like to know if the word "x" in $file1, is present in the sentence "y" in file2. My files have more than 20000 lines..

Example:

This is the content of the first file :

eat
take
breath
you
alpha

This is the content of the second file :

eat,hungry
love,lovers
me,mine
take,taken,give
you,u,yo
fun,funny

This is the content that I might expect for the third file

eat : eat,hungry
take : take,taken,give
you : you,u,yo

So as you can see, I would like to find the matching expression in the second file of the word of the first file.

My solutions - but the loop never ends -

solution1:

$file1= "words.txt";
$file2 = "expressions.txt";
$out = "out.txt";

open (W, "<", $file1);
open (E, "<", $file2);
open (OUT, ">", $out);

while(defined($l = <W>)){
    @a = split (/\n/, $l);
    push @w, @a;
}

while(defined($l2 = <E>)){
    for ($i = 0; $i < @w; $i++){
        if (grep /\Q\b$w[$i]\b\E/, $l2){ #or just /\b$w[$i]\b/
            print OUT "$w[$i] : $l2\n";
        }
    }
}

solution2:

$file1= "words.txt";
$file2 = "expressions.txt";
$out = "out.txt";

open (W, "<", $file1);
open (E, "<", $file2);
open (OUT, ">", $out);

while(defined($l = <W>)){
    @a = split (/\n/, $l);
    push @w, @a;

while(defined($l2 = <E>)){
    @b = split (/\n/, $l2);
    push @e, @b;
}

for ($k = 0; $k < @e; $k++){
    for ($i = 0; $i < @w; $i++){
        if (grep /\b$w[$i]\b/, $e[$k]){
            print OUT "$w[$i] : $w[$l]\n";
        }
    }
}
3
  • Sidenote, $l2 =~ /\Q\b$w[$i]\b\E/ is better than grep /\Q\b$w[$i]\b\E/, $l2 and if you want to match word boundary and NOT literal \b then /\b\Q$w[$i]\E\b/
    – mpapec
    Jun 20, 2014 at 9:03
  • Ok I edited. However, why is better the first expression ?
    – KeyPi
    Jun 20, 2014 at 9:13
  • $l2 =~ .. is better as it better communicates intention, which is to match regex against single string.
    – mpapec
    Jun 20, 2014 at 11:18

3 Answers 3

1

How about process the expression file first to make a dictionary map every word to a sentence, then find if the word in words.txt is in the dictionary? I guess this may be faster. Source code is as below:

#! /opt/VRTSperl/bin/perl

    $words = "words.txt";
    $expressions = "expressions.txt";
    $out = "out.txt";

    open (E, "<", $expressions);
    open (W, "<", $words);
    open (OUT, ">", $out);

    my %dic;

    while (my $sentence = <E>) {
        chomp($sentence);
        my @words = split(/,/, $sentence);
        foreach my $word (@words) {
            $dic{$word} .= "$sentence";
        }   
    }

    while (my $word = <W>) {
        chomp($word);
        if ($dic{$word}) {
            print OUT "$word : $dic{$word}\n"
        }   
    }
1
  • really? in my side, the out file is eat : eat,hungry take : take,taken,give you : you,u,yo
    – cartman
    Jun 21, 2014 at 6:12
1
#!/usr/local/bin/perl
use strict;
use warnings;
open (my $fh, "<", "f1.txt") or die $!;
open (my $fh2, "<", "f2.txt") or die $!;
my @keys;
while(chomp(my $line = <$fh>)){
        push @keys, $line;
}
while(chomp(my $line2 = <$fh2>)){
        foreach (@keys){
                if ($line2 =~ $_){
                        print "$_ : $line2\n";
                }
        }
}

Output

eat : eat,hungry
take : take,taken,give
you : you,u,yo
4
  • will it work if, for ex., the word is in the middle? for ex. the word is "a", and the expression is "c,a,b".
    – KeyPi
    Jun 20, 2014 at 15:31
  • after 10 min is still running . However I've already noticed some errors -> continue : continuellement, constamment
    – KeyPi
    Jun 20, 2014 at 15:41
  • Sorry I don't get it, please rephrase your question by editing it, I'd be happy to help. Jun 20, 2014 at 16:22
  • sorry! for ex. in file 1 there is the word "continue", in file 2 there is "continuellement, constamment"... So it seems that for it "continue" is the same of "continuellement" so I think it doesn't recognized the right word. The other question is : if the word "continue" is in the middle of the expression of file 2 (so for ex. "continuellement, CONTINUE, continuel" I think your code doesn't work. Did I explain better the question?
    – KeyPi
    Jun 20, 2014 at 16:28
1

You're trying to match literal \b instead of word boundary, so

/\Q\b$w[$i]\b\E/

should actually be

/\b\Q$w[$i]\E\b/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.