1

Since this morning, my website has started failing with 403 error codes when anybody attempts to share a link to my website.

When I use the Facebook debugger tool, it is suggesting 'This response code could be due to a robots.txt block. Please allowlist facebookexternalhit on your sites robots.txt config to utilize Facebook scraping'

https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Fwww.infinitesweeps.com%2Fsweepstake%2F280358-Sierra-Nevada-Game-Day-Giveaway.html

I have a whitelist override rule setup in Cloudflare and show the Facebook bot coming to the page every time I attempt to update the last scraped time and receiving a '200 OK' response from a Cached Page (meaning that Cloudflare is sending Facebook a Cached page of the link).

But for some reason, Facebook is saying 'The document returned no data.'

https://developers.facebook.com/tools/debug/echo/?q=https%3A%2F%2Fwww.infinitesweeps.com%2Fsweepstake%2F280358-Sierra-Nevada-Game-Day-Giveaway.htmlopers

All of my 403 pages for the website return data (an HTML page saying that you re blocked or error etc). Basically, nothing should return blank.

Is this happening for anybody else? I have tried debugging it all morning and I am sure that Facebook bot is correctly accessing my website (receiving a 200 OK from a Cached page that other users are also receiving fine) - yet it's showing these errors.

Where can I begin to fix this or how?

I have updated my robots.txt file to specifically allow the Facebook bot (and Google bot and Twitter).

https://www.infinitesweeps.com/robots.txt

1
  • Looks like Meta/FB made some changes and the rule is now required in robots.txt.
    – Mr Pablo
    Commented Sep 2 at 14:28

1 Answer 1

0

The issue resolved it's self for me after almost 12 hours. The only thing I really did was update my robots.txt file to include:

User-agent: Twitterbot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: *
Disallow: /upgrade/
Disallow: /share/
Disallow: /twitter/
Disallow: /facebook/
Disallow: /sponsor/
Disallow: /paypal/
Disallow: /forum/profile/
Disallow: /login.php?*
Allow: /

I previously had

User-agent: *
Disallow: /upgrade/
Disallow: /share/
Disallow: /twitter/
Disallow: /facebook/
Disallow: /sponsor/
Disallow: /paypal/
Disallow: /forum/profile/
Disallow: /login.php?*

Hope this works for you if you're having the same problem! Took about 1 hour.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.