1

Recently I noticed that Facebook's Object Debugger was unable to scrape any pages of my website. After troubleshooting and scouring the internet, I'm at a loss for what might be causing this bug.

Whenever I attempt to fetch a new scrape of my website, the following error is returned:

Error parsing input URL, no data was cached, or no data was scraped.

When clicking into "See exactly what our scraper sees for your URL", the scraper returns:

Document returned no data

This is obviously a bit difficult to debug given the lack of data. Here's what I've tried thus far:

  • Checked DNS settings, everything seems fine
  • Tried using "Fetch as Google," GoogleBot had no problem fetching the page HTML
  • Verified all the meta settings on the site. fb:app_id, og:title, og:description, og:site_name, og:url, and og:type are all present.
  • Made sure canonical URL references the homepage, and does not have any trailing slash or trailing data.
  • Rolled back commits to before the last successful crawl date

I'm at a loss for what could be causing this. If anyone has any ideas, or needs more information, I would be happy to provide it.

After checking access logs, I see the following:

173.252.112.115 - - [22/Jun/2015:20:49:02 +0000] "GET / HTTP/1.1" 404 993 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

But this is strange, as it is immediately followed by a normal user:

[user ip] - - [22/Jun/2015:20:48:09 +0000] "GET / HTTP/1.1" 200 28227 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7"

There is nothing in robots.txt to disallow bots.

EDIT: This site is running on Django, and AngularJS is serving my pages. I'm using django-seo-js to work with prerender to improve SEO.

5
  • Can you give as the URL to an actual page? Have you checked your server’s access and error logs, for the timestamp where you triggered a scrape? Does the request from the FB scraper even reach your server?
    – CBroe
    Jun 22, 2015 at 20:43
  • The url is taggler.com ; I'm going to look into the access logs and I'll report back Jun 22, 2015 at 20:45
  • It appears to register in my access log when I request a scrape. 173.252.112.115 - - [22/Jun/2015:20:49:02 +0000] "GET / HTTP/1.1" 404 993 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" Looks like it's hitting a 404? We have some logs that indicate there were also 200 responses. Jun 22, 2015 at 20:49
  • 1
    Yes, when I visit your page in Chrome and send facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) as value of the User-Agent header, I get a 404 as well (I used ModHeader extension for that), whereas requests with my normal Chrome User-Agent show me your start page just fine. So investigate if you have any plugins, “firewall” rules or similar set up to fight requests by “bots” – might be something is a little overzealous in that regard when it comes to visits from the FB scraper.
    – CBroe
    Jun 22, 2015 at 20:55
  • Yeah...it's throwing a Javascript stack trace. This seems like it might be getting caused by prerender. Thanks for the help - feel free to post your response and I can accept it as the answer. Jun 22, 2015 at 21:09

1 Answer 1

1

When I visit your page in Chrome and send facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) as value of the User-Agent header, I get a 404 as well (I used ModHeader extension for that), whereas requests with my normal Chrome User-Agent show me your start page just fine.

So investigate if you have any plugins, “firewall” rules or similar set up to fight requests by “bots” – might be something is a little overzealous in that regard when it comes to visits from the FB scraper.
That doesn’t seem to be it though (was an educated guess only, since this is often the cause of such problems), but as you said,

it's throwing a Javascript stack trace. This seems like it might be getting caused by prerender

– let us know if you found the exact cause.

3
  • 1
    This is indeed the case. Facebook bot's user-agent is sent to the our ?_escaped_fragment_= URL, which is in turn throwing an error with prerender.io. A ticket is in to prerender to determine the cause of this 404. Jun 22, 2015 at 21:34
  • 2
    As it turns out, this was caused by a faulty nginx server. After rebuilding a new server, everything is back to normal. Jun 22, 2015 at 23:14
  • 4
    @ZackTanner can you elaborate on that point, I think I have the same issue.
    – justberare
    Dec 12, 2015 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.