7

Currently I am using simple_html_dom to scrape a website view here to see the website I am scraping, everything comes back fine except it continues to put the same content for every single post it scrapes.. View here to see demo

$page = (isset($_GET['p'])&&$_GET['p']!=0) ? (int) $_GET['p'] : '';  
$html = file_get_html('http://screenrant.com/movie-news/'.$page);

foreach($html->find('#site-top > div.site-wrapper > div.top-content > article > section > ul > li > div.info > h2 > a') as $element)
{
    print '<br><br>';
    echo $url = ''.$element->href;
    $html2 = file_get_html($url);

    $image = $html2->find('meta[property=og:image]',0);
    $news['image'] = $image->content;
    #print '<br><br>';

    // Ending The Featured Image

    #site-top > div.site-wrapper > div.top-content > article > section > ul > li:nth-child(2)

    $title = $html2->find('#site-top > div.site-wrapper > div.top-content > article > header.single-header > h1',0);
    $news['title'] = $title->plaintext;

    // Ending the titles
    print '<br>';
    #site-top > div.site-wrapper > div.top-content > article > div
    $articles = $html2->find('#site-top > div.site-wrapper > div.top-content > article > div > p');
    foreach ($articles as $article) {
    #echo "$article->plaintext<p>"; 
    $news['content'] = $news['content'] . $article->plaintext . "<p>";
    }

    print '<pre>';print_r($news);print '</pre>';

    print '<br><br>';

        // mysqli_query($DB,"INSERT INTO `wp_scraped_news` SET
             //                   `hash` = '".$news['title']."',
               //                 `title` = '".$news['title']."',
                 //               `image` = '".$news['image']."',
                   //             `content` = '".$news['content']."'");
         // print '<pre>';print_r($news);print '</pre>';
}

I have no idea where I am going wrong here but I am assuming it's one of two things and I have messed around with both of these things with no luck.

1. I am doing something wrong with how my foreach are laid out.

2. The website is changing selectors for each new article.

In both cases I am probably wrong.. but I've tinkered with them both for about 2 hours now and at the point of giving up.. any help is very appreciated.

2
  • I think your select path may be wrong. In Chrome, select the HTML element you want, right-click then click Copy > Copy selector. I get something that looks like this: #site-top > div.site-wrapper > div.top-content > article > section > ul > li:nth-child(2) > div.info > div > div > p -- probably need to change this li:nth-child(2) to this li .. a place to start though.
    – Clay
    Dec 24, 2015 at 2:21
  • 3
    @Clayton if you look at the code displayed, I have that commented out, because I've tried it already.. - Just put that code in again, and it returns only the snippets displayed on the News archive page, we're declaring the URL at the top which means we're able to scrape the content pages also. Dec 24, 2015 at 2:23

1 Answer 1

4

The problem is that you're not clearing out the old content from $news['content']. So when you process the second page, you're appending its content to the content of the first page. And the third page appends to this again, and so on.

Put

$news['content'] = '';

before

foreach ($articles as $article) {
6
  • Look at your output more carefully. It's not showing the same content for each post. The beginning is the same, but it gets longer each time.
    – Barmar
    Dec 24, 2015 at 2:40
  • I am for sure doing this incorrectly.. can you please edit your post to display exactly how I should have it? Because placing $article['content'] = ''; above my foreach ($articles as $article) { just returns a error on line 43. Dec 24, 2015 at 2:41
  • Sorry, that was a typo. It should be $news['content'] = '';. That's the variable I said you need to clear out in the text.
    – Barmar
    Dec 24, 2015 at 2:42
  • Didn't you think it was strange that I said you need to clear out one variable, then my code said to clear a different variable? If you thought about it, it should have been obvious that I made a mistake.
    – Barmar
    Dec 24, 2015 at 2:44
  • 1
    Don't just blindly copy code, understand it. Otherwise you don't learn.
    – Barmar
    Dec 24, 2015 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.