2

I'm trying to extract quite a bit of data from a perfectly structured web page and struggling with Mojo::DOM methods. I would really appreciate it if anyone could point me in the right direction.

The truncated HTML with interesting data follows:

 <div class="post" data-story-id="3964117" data-visited="false">//extracting story-id
  <h2 class="post_title page_title"><a href="http://example.com/story/some_url" class="to-comments">header.</a></h2>
  //useless data and tags

<a href="http://example.com/story/some_url" class="b-story__show-all">
  <span>useless data</span>
</a>

<div class="post_tags">
  <ul>
    <li class="post_tag post_tag_strawberry hidden"><a href="http://example.com/search.php?n=32&r=3">&nbsp;</a></li>
    <li class="post_tag"><a href="http://example.com/tag/tag1/hot">tag1</a></li>
    <li class="post_tag"><a href="http://example.com/tag/tag2/hot">tag2</a></li>
    <li class="post_tag"><a href="http://example.com/tag/tag1/hot">tag3</a></li>
  </ul>
</div>

<div class="post_actions_box">

  <div class="post_rating_box">
    <ul data-story-id="3964117" data-vote="0" data-can-vote="true">
      <li><span class="post_rating post_rating_up control">&nbsp;</span></li>
      <li><span class="post_rating_count control label">1956</span></li> //1956 - interesting value
      <li><span class="post_rating post_rating_down control">&nbsp;</span></li>
    </ul>
  </div>

  <div class="post_more_box">
    <ul>
      <li>
        <span class="post_more control">&nbsp;</span>
      </li>
      <li>
        <a class="post_comments_count label to-comments" href="http://example.com/story/some_url#comments">132&nbsp;<i>&nbsp;</i></a>
      </li>
    </ul>
  </div>

</div>
</div>

What I have right now is

use strict;
use warnings;

use Data::Dumper;
use Mojo::DOM;


my $file = "index2.html";
local( $/, *FH ) ;
open( FH, $file ) or die "sudden flaming death\n";
my $text = <FH>;
my $dom = Mojo::DOM->new;
$dom->parse($text);
my $ids = $dom->find('div.post')
    ->each (sub {print $_->attr('data-story-id'), "\n";});
$dom->find('a.to-comments')->each (sub {print $_->text, "\n";});

This mess extracts data-story-id from the src and header value (tested the same with href value), but all my other attempts fail.

3964117
Header
132

"post_rating_count control label" is not extracted. I could get the first href values with searching for a.to-comments and returning attr('href'), but for some reason it also returnes me values of a link in the end of the segment with class="post_comments_count label to-comments". The same happens with header value extraction.

In the final end I am looking for an array with data structure with following fields:

  • story-id (this is is a success)
  • href (somehow, matching more than needed.)
  • header (somehow, matching more than needed.)
  • list of tags as a string (no idea how to do that)

What is more, I feel it is possible to optimize the code and make it look a bit better, but my kung-fu is not so strong.

8
  • Your HTML isn't well-formed. You have a superfluous closing </div> tag, which makes me wonder whether you have removed an opening <div> tag in the "useless content and html structures". Please can you fix it
    – Borodin
    Feb 3, 2016 at 9:29
  • yes, indeed, sorry about that. updated the code.. Feb 3, 2016 at 9:57
  • Thank you. Is there not a single element that encloses all the data for each post? In other words, does the next <div class="post"> really follow on immediately after the <div class="post_actions_box">?
    – Borodin
    Feb 3, 2016 at 10:08
  • Just was in progress for testing. May I please ask you several questions in the "answer" section? Feb 3, 2016 at 23:46
  • I'm not happy with my solution until you've resolved my doubts about the real content of the HTML, so can we fix that first? Stack Overflow is a knowledge base rather than a forum, so every entry is like a Wikipedia article and should be as correct as possible. I can create a chat room where we can talk, but I would like to know the nature of your questions first
    – Borodin
    Feb 3, 2016 at 23:54

1 Answer 1

3

Your HTML is malformed as I said in my comment. I've guessed where the missing <div> might go but I'm probably wrong. I've assumed the last </div> in the data corresponds to the first <div>, so that the whole block constitutes a single post

The main problem you have is trying to do everything inside an each method call on your Mojo::Collection objects. It's far easier to use Perl to iterate of each collection, like this

use strict;
use warnings;

use Mojo::DOM;

use constant HTML_FILE => 'index2.html';

my $html = do {
    open my $fh, '<', HTML_FILE or die $!;
    local $/;
    <$fh>;
};

my $dom = Mojo::DOM->new($html);

for my $post ( $dom->find('div.post')->each ) {

    printf "Post ID:     %s\n", $post->attr('data-story-id');

    my $anchor = $post->at('h2.post_title > a');
    printf "Post href:   %s\n", $anchor->attr('href');
    printf "Post header: %s\n", $anchor->text;

    my @tags = $post->find('li.post_tag > a')->map('text')->each;

    printf "Tags:        %s\n", join ', ', @tags;

    print "\n";
}

output

Post ID:     3964117
Post href:   http://example.com/story/some_url
Post header: Header
Tags:        some_value, tag1, tag2, tag3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.