0

Strange case, Yandex bot is massively overloading my website. The problem is mine obviously, as I have some ajax filters on website active but quite heavy if they are called altogether like bots do.

I have tried with many robots.txt, but they are having no effect. The kind of URL needed to be blocked are in this following form:

 /de/component/customfilters/0-zu-15-eur/nKein+Herstellerf.html?custom_f_116[0]=38&custom_f_116[1]=4c&custom_f_116[2]=39&start=100

But they are URL rewrited not physical. The physycal folder is already blocked in robotx.txt

How can solve this and how can check if Yandex bot are not reading the robots.txt file?

Every time I edit robots.txt file should I restart Apache? I think no like htaccess

5
  • 1
    You do not need to restart Apache when changing your robots.txt file. It's just a regular web file. Not a config file or anything else related to the web server.
    – John Conde
    Sep 17, 2014 at 14:25
  • 1
    You can't really block bad bots by robots.txt, they're mostly ignoring it.
    – user1299518
    Sep 17, 2014 at 14:28
  • Can’t you “block” everything from a certain base URL, say f.e. /de/component/customfilters/? If all your heavy AJAX requests go below that path, but not anything else that is supposed to still be accessible to search engines … then simply say you don’t want anything below that path to be crawled/indexed. And/or you might want to tell the Yandex bot to slow down and keep a certain pause between requests – a quick search suggests that should be easily possible as well.
    – CBroe
    Sep 17, 2014 at 14:30
  • Thanks all for replying. Ok for robots to not be update.
    – giuseppe
    Sep 17, 2014 at 14:33
  • how I can tell yandex to slow down keeping pauses between requests? I was thinking about blocking in php code, taking the user agent. But how I can know the correct user agent?
    – giuseppe
    Sep 17, 2014 at 14:34

1 Answer 1

3

Mitigate the current heavy load

If your website is currently under heavy load from this crawler, it is possible that making the appropriate changes to your robots.txt won't actually help right now. The lovely people of the Yandex dev team do claim that their bots will visit robots.txt before it crawls - but I think that, if the crawl has started, it may not read any changes until the next time it wants to crawl. They may also have a cached copy of your robots.txt from before you changed it. You can take a look in your server logs to see if they've visited robots.txt since it was changed. My guess is probably not.

There's also the possibility that a bad bot is pretending to be Yandex while crawling your site. Bad bots usually ignore the robots.txt rules anyway. So any changes you make may affect Yandex correctly, but not the bad bots.

In either case, if this crawler is putting your server under heavy load now, then you'll want to block them now and decide later if you want to make that a temporary or permanent block.

One way to do this is using the BRowserMatchNoCase directive in .htacccess:

BrowserMatchNoCase "Yandex" bots

Order Allow,Deny
Allow from ALL
Deny from env=bots

Or, you could use a rewrite rule in .htaccess, instead:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(Yandex).*$ [NC]
RewriteRule .* - [F,L]

As for your robots.txt

It doesn't matter whether the URL is being rewritten or not, the bot will crawl any URLs it finds unless you disallow that URL. If you're disallowing the physical folder and the URL doesn't point to that folder, then the Disallow won't work.

Try something like this in your robots.txt:

Disallow: /de/component/customfilters/

This will ask all bots not to crawl any urls that contain /de/component/customfilters/. If you only want to talk to Yandex bots, you can specify that, too:

User-agent: Yandex  # directives after this line will only apply to Yandex bots.
Disallow: /de/component/customfilters/

If you want to check that Yandex are reading your robots.txt, they have a test tool here:

http://webmaster.yandex.ru/robots.xml (the page is in Russian)

If you just want Yandex to slow down, you can add a crawl delay directive for Yandex bots:

User-agent: Yandex # directives after this line will only apply to Yandex bots.
Crawl-delay: 2 # specifies a delay of 2 seconds 

More information: https://help.yandex.com/webmaster/controlling-robot/robots-txt.xml#crawl-delay

4
  • Thanks for your answer. yes I want to disable those links to all bots. I have tried putting Disallow: /de/component/customfilters/ in robots.txt but it didn't work
    – giuseppe
    Sep 17, 2014 at 14:46
  • There are a couple of additional considerations to consider if your site is currently under load from the bot; is this really who it claims to be? Have they read the robots.txt file since you've changed it? Ultimately, if they're causing an unacceptable load on your website, then you'll want to block them, at least temporarily. I'll update my answer to elaborate.
    – 0x00h
    Sep 17, 2014 at 18:13
  • Nice question, but I don't know. I have taken their IP, and do a check , they are indexed in ip like Yandex bot. From their behaviour it should be so, because they are crawling all website, so i suppose yes
    – giuseppe
    Sep 18, 2014 at 7:30
  • If they are harming your website, you need to stop them. It's possible that they have not seen your changes to robots.txt. Take a look at my updated answer for tips on blocking them. After blocking them you can make a decision if you want to let them try again, or not.
    – 0x00h
    Sep 18, 2014 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.