4

Assuming that I have an HTML page as follows:

<!-- This is the opening tag -->
<div class="content_text">
  <div>Title</div>
  <div>Author Name</div>
  <div>Some complicated HTML elements correctly validated</div>
  <b>Some more text</b>
  <img ... />
  <div> more and more text </div>
</div><!-- This is the correct closing tag -->

How do I get the content between the opening of the div with class="content_text" and its correct closing tag?

I tried regular expressions, but I couldn't find any easy or even hard way to do it.

I tried XPath, but I still couldn't get the content. Instead I got the text inside the outer div.

10
  • Not sure of your specific application.. but any chance of using jQuery? Its html method seems like it would be useful.
    – showdev
    Apr 9, 2013 at 22:09
  • No, it's a server-side script
    – Shehabic
    Apr 9, 2013 at 22:10
  • @showdev jQuery is clientside, and since the OP tagged this as PHP, I don't think he'll need it.
    – HamZa
    Apr 9, 2013 at 22:10
  • @Shehabix could you provide a simplified sample and the code you tried ?
    – HamZa
    Apr 9, 2013 at 22:12
  • 1
    Not the right regex, but it works for this example preg_match_all('/<div class="content_text">(\s*<.*?>.*?<\/div>\s*|\s*<.*? \/>\s*)<\/div>/s', $html, $m);, but to make it clear just use DOM !
    – HamZa
    Apr 9, 2013 at 22:25

4 Answers 4

5

You can use the PHP Simple HTML DOM Parser to parse HTML like DOMDocument would for XML.

Note: PHP has support for DOMDocument directly as well.

8
  • That parser has been written for PHP 4, PHP 5 ships with its own HTML parser called DOMDocument the method to load HTML is called loadHTML.
    – hakre
    Apr 17, 2013 at 15:40
  • @hakre, This parser requires PHP5 to work. And it explicitly says it was written for PHP5.
    – Shoe
    Apr 17, 2013 at 15:41
  • Yes it does, but it has been written for PHP 4. Since PHP 5 the need was gone. Sure the lib was ported to PHP 5 but I would say it has overlived its time because the pre-condition for what it has been created (PHP w/o HTML parser) is not the situation any longer (PHP is with HTML parser).
    – hakre
    Apr 17, 2013 at 15:43
  • @hakre, so you have downvoted this answer because I suggested a library that you consider to have overlived its time but totally works and fits in the requirements of this question?
    – Shoe
    Apr 17, 2013 at 15:47
  • No because the answer is (more) wrong. The suggested library does not follow the DOM Specification for HTML documents nor follows it the rules of CSS selectors (not even for the subset it says it supports), nor does it properly parse HTML.
    – hakre
    Apr 17, 2013 at 15:51
4
    $scrape_address = "http://www.al-madina.com/node/444862";
    $ch = curl_init($scrape_address);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, '1'); 
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_ENCODING, "");
    $data = curl_exec($ch);
    // I couldn't get an element by Attribute so I just replaced class to id
    $data = str_replace('class="content_text"','id="my_unique_id"',$data);

    $domd = new DOMDocument();
    libxml_use_internal_errors(true);
    $domd->loadHTML($data);
    libxml_use_internal_errors(false);
    $div = $domd->getElementById("my_unique_id");

    if ($div) {
      $dom2 = new DOMDocument();
      $dom2->appendChild($dom2->importNode($div, true));
      echo $dom2->saveHTML();
    } else {
      echo "Nothing found";
    }
2

I'd suggest PHP's DOMDocument - unless the content will be always structured exactly the same way, regular expressions won't do, and even then it's not going to be pretty.

Also, here is a question about a similar situation that was solved by using SimpleXML, maybe that can help.

3
  • "The DOM extension allows you to operate on XML documents". It's for XML.
    – Shoe
    Apr 9, 2013 at 22:23
  • 1
    DOMDocument works with HTML, too, e.g., php.net/manual/en/domdocument.loadhtml.php Apr 9, 2013 at 22:24
  • I found the solution to my situation and posted it already few minutes ago, using DOMDocument
    – Shehabic
    Apr 9, 2013 at 22:26
0

You already seem to be able to successfully run XPath queries, so I'm omitting PHP code and get straight to the XPath part.

Not sure what you mean by "content", so I'm offering some alternatives:

You want all text nodes inside the <div/>:

//div[@class="content_text"]//text()

You want all XML including the elements:

//div[@class="content_text"]

Both will return a set of results, be sure to loop over it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.