1

I have some data that I need to analyze. The data is multilined and each block is separated by a newline. So, it is something like this

Property 1: 1234
Property 2: 34546
Property 3: ACBGD

Property 1: 1234
Property 4: 4567

Property 1: just
Property 3: an
Property 5: simple
Property 6: example

I need to filter out those data blocks that have some particular Property present. For example, only those that have Property 4, only those that have Property 3 and 6 both etc. I might also need to choose based upon the value at these Properties, so for example only those blocks that have Property 3 and its value is 'an'.

How would I do this in Perl. I tried splitting it by "\n" but didn't seem to work properly. Am I missing something?

1
  • 3
    Can you post what code you have so far, so we can see what you are trying?
    – B Johnson
    Nov 4, 2010 at 8:57

8 Answers 8

14

The secret to making this task simple is to use the $/ variable to put Perl into "paragraph mode". That makes it easy to process your records one at a time. You can then filter them with something like grep.

#!/usr/bin/perl

use strict;
use warnings;

my @data = do {
  local $/ = '';
  <DATA>;
};

my @with_4   = grep { /^Property 4:/m } @data;

my @with_3   = grep { /^Property 3:/m } @data;
my @with_3_6 = grep { /^Property 6:/m } @with_3;

print scalar @with_3_6;

__DATA__
Property 1: 1234
Property 2: 34546
Property 3: ACBGD

Property 1: 1234
Property 4: 4567

Property 1: just
Property 3: an
Property 5: simple
Property 6: example

In that example I'm processing each record as plain text. For more complex work, I'd probably turn each record into a hash.

#!/usr/bin/perl

use strict;
use warnings;

use Data::Dumper;

my @data;

{
  local $/ = '';

  while (<DATA>) {
    chomp;

    my @rec = split /\n/;
    my %prop;
    foreach my $r (@rec) {
      my ($k, $v) = split /:\s+/, $r;
      $prop{$k} = $v;
    }

    push @data, \%prop;
  }
}

my @with_4   = grep { exists $_->{'Property 4'} } @data;

my @with_3_6 = grep { exists $_->{'Property 3'} and
                      exists $_->{'Property 6'} } @data;

my @with_3an = grep { exists $_->{'Property 3'} and
                      $_->{'Property 3'} eq 'an' } @data;

print Dumper @with_3an;

__DATA__
Property 1: 1234
Property 2: 34546
Property 3: ACBGD

Property 1: 1234
Property 4: 4567

Property 1: just
Property 3: an
Property 5: simple
Property 6: example
6
  • @davorg thanx a lot, this really seems to work. but one thing I noticed is that in the final array the order is not maintained as in the original data, its not an issue here but was wondering if that could be maintained in some way as well.
    – sfactor
    Nov 4, 2010 at 10:35
  • @sfactor: My script feeds your data into a list, which preserves the order. Nov 4, 2010 at 10:56
  • If you wanted to get the hashes printed out in the same order as the original data, then in this case you could simply sort the keys. In a more complex example, you could store another value somewhere in the data structure which defines the order. I'm loathe to switch to using an array to store the individual records as hash look-ups are so useful here.
    – Dave Cross
    Nov 4, 2010 at 11:07
  • @Alex Reynolds: Your program still stores the individual records as hashes. So your data is shuffled up too - the dumper output shows that the keys aren't in numerical order.
    – Dave Cross
    Nov 4, 2010 at 11:10
  • 2
    Right. Your data structure is almost identical to mine. The records are stored in an array (so they remain in the original order) but the values within each record are stored in a hash so the the order is lost. That's exactly the same as my solution. The OP is clear that the ordering of those values isn't important in this case. He was just asking if ordering could be preserved if necessary. Both of our solutions would require very similar changes to support that.
    – Dave Cross
    Nov 4, 2010 at 11:36
3

Dependent on the size of each property set and how much memory you have...

I'd use a simple state machine that scans sequentially through the file - with a line-by-line sequential scan, not multiline - adding each property/id/value to a hash keyed on id. When you get a blank line or end-of-file, determine whether the elements of the hash should be filtered in or out, and emit them as necessary, then reset the hash.

2

Quick and dirty:

my $string = <<END;
Property 1: 1234
Property 2: 34546
Property 3: ACBGD

Property 1: 1234
Property 4: 4567

Property 1: just
Property 3: an
Property 5: simple
Property 6: example
END

my @blocks = split /\n\n/, $string;

my @desired_blocks = grep /Property 1: 1234/, @blocks;

print join("\n----\n", @desired_blocks), "\n";
2
#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;

my $propertyRef;
my $propertyRefIdx = 0;

while (<>) {
    chomp($_);
    if ($_ =~ /Property (\d+): (.*)/) {
        my $propertyKey = $1;
        my $propertyValue = $2;

        $propertyRef->[$propertyRefIdx]->{$propertyKey} = $propertyValue;
    }
    else {
        $propertyRefIdx++;
    }
}

print Dumper $propertyRef;

Let's say this script is called propertyParser.pl and you have a file containing the properties and values called properties.txt. You could call this as follows:

$ propertyParser.pl < properties.txt

Once you have populated $propertyRef with all your data, you can then loop through elements and filter them based on whatever rules you need to apply, such as certain key and/or value combinations:

foreach my $property (@{$propertyRef}) {
    if (defined $property->{1} && defined $property->{3} 
                               && ! defined $property->{6}) {
        # do something for keys 1 and 3 but not 6, etc.
    }
}
1
  • I like this. It would be simple to add on a CLI to specify properties and values to be matches. Nice and clean.
    – sorpigal
    Nov 4, 2010 at 12:28
1

Your record separator should be "\n\n". Every line ends with one, and you differentiate a block by a double newline. Using this idea, it was rather easy to filter out the blocks with Property 4.

use strict;
use warnings;
use English qw<$RS>;

open( my $inh, ... ) or die "I'm dead!";

local $RS = "\n\n";
while ( my $block = <$inh> ) { 
    if ( my ( $prop4 ) = $block =~ m/^Property 4:\s+(.*)/m ) { 
        ...
    }
    if ( my ( $prop3, $prop6 ) 
             = $block =~ m/
        ^Property \s+ 3: \s+ ([^\n]*)
        .*?
        ^Property \s+ 6: \s+ ([^\n]*)
        /smx 
       ) {
        ...
    }
}

Both expressions use a multiline ('m') flag, so that ^ applies to any line start. The last one uses the flag to include newlines in '.' expressions ('s') and the extended syntax ('x') which, among other things, ignores whitespace within the expression.

If the data was rather small, you could process it all in one go like:

use strict;
use warnings;
use English qw<$RS>;

local $RS = "\n\n";
my @block
    = map { { m/^Property \s+ (\d+): \s+ (.*?\S) \s+/gmx } } <DATA>
    ;
print Data::Dumper->Dump( [ \@block ], [ '*block' ] ), "\n";

Which shows the result to be:

@block = (
           {
             '1' => '1234',
             '3' => 'ACBGD',
             '2' => '34546'
           },
           {
             '4' => '4567',
             '1' => '1234'
           },
           {
             '6' => 'example',
             '1' => 'just',
             '3' => 'an',
             '5' => 'simple'
           }
         );
0

Check what the $/ variable will do for you, for example explanation here. You can set the 'end of line' separator to be whatever you please. You could try setting it to '\n\n'

$/ = "\n\n";
foreach my $property (<DATA>)
    {
    print "$property\n";
    }


__DATA__
Property 1: 1234
Property 2: 34546
Property 3: ACBGD

Property 1: 1234
Property 4: 4567

Property 1: just
Property 3: an
Property 5: simple
Property 6: example

As your data elements seem to be deilmited by blank lines this will read each property group of lines one by one.

You could also read the entire file into an array and process it from memory

my(@lines) = <DATA>

1
  • 4
    A few comments here. It is safer to local ly modify $/ rather than globally. And the so-called 'paragraph' slurp mode can be enabled by setting $/ to ''.
    – Zaid
    Nov 4, 2010 at 9:58
0

Assuming that your data are stored into a file (let's say mydata.txt), you could write the following perl script (let's call him Bob.pl):

my @currentBlock = ();
my $displayCurrentBlock = 0;
# This will iterate on each line of the file
while (<>) {
  # We check the content of $_ (the current line)
  if ($_ =~ /^\s*$/) {
    # $_ is an empty line, so we display the current block if needed
    print @currentBlock if $displayCurrentBlock;
    # Current block and display status are resetted
    @currentBlock = ();
    $displayCurrentBlock = 0;
  } else{
    # $_ is not an empty line, we add it to the current block
    push @currentBlock, $_;
    # We set the display status to true if a certain condition is met
    $displayCurrentBlock = 1 if ($_ =~ /Property 3: an\s+$/);
  }
}
# A last check and print for the last block
print @currentBlock if $displayCurrentBlock;

Next, you just have to lauch perl Bob.pl < mydata.txt, and voila !

localhost> perl Bob.pl < mydata.txt
Property 1: just
Property 3: an
Property 5: simple
Property 6: example
1
  • 4
    Never, ever, ever write $_ =~ /.../!!
    – tchrist
    Nov 4, 2010 at 13:46
0

In relation to the first part of your question, you can read records in "paragraph mode" using perl's -00 commandline option, for example:

#!/usr/bin/perl -00

my @data = <>;

# Print the last block.
print $data[-1], "\n"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.