Mitigate the current heavy load
If your website is currently under heavy load from this crawler, it is possible that making the appropriate changes to your robots.txt won't actually help right now. The lovely people of the Yandex dev team do claim that their bots will visit robots.txt before it crawls - but I think that, if the crawl has started, it may not read any changes until the next time it wants to crawl. They may also have a cached copy of your robots.txt from before you changed it. You can take a look in your server logs to see if they've visited robots.txt since it was changed. My guess is probably not.
There's also the possibility that a bad bot is pretending to be Yandex while crawling your site. Bad bots usually ignore the robots.txt rules anyway. So any changes you make may affect Yandex correctly, but not the bad bots.
In either case, if this crawler is putting your server under heavy load now, then you'll want to block them now and decide later if you want to make that a temporary or permanent block.
One way to do this is using the BRowserMatchNoCase directive in .htacccess
:
BrowserMatchNoCase "Yandex" bots
Order Allow,Deny
Allow from ALL
Deny from env=bots
Or, you could use a rewrite rule in .htaccess
, instead:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(Yandex).*$ [NC]
RewriteRule .* - [F,L]
As for your robots.txt
It doesn't matter whether the URL is being rewritten or not, the bot will crawl any URLs it finds unless you disallow that URL. If you're disallowing the physical folder and the URL doesn't point to that folder, then the Disallow won't work.
Try something like this in your robots.txt
:
Disallow: /de/component/customfilters/
This will ask all bots not to crawl any urls that contain /de/component/customfilters/
.
If you only want to talk to Yandex bots, you can specify that, too:
User-agent: Yandex # directives after this line will only apply to Yandex bots.
Disallow: /de/component/customfilters/
If you want to check that Yandex are reading your robots.txt, they have a test tool here:
http://webmaster.yandex.ru/robots.xml
(the page is in Russian)
If you just want Yandex to slow down, you can add a crawl delay directive for Yandex bots:
User-agent: Yandex # directives after this line will only apply to Yandex bots.
Crawl-delay: 2 # specifies a delay of 2 seconds
More information: https://help.yandex.com/webmaster/controlling-robot/robots-txt.xml#crawl-delay
robots.txt
, they're mostly ignoring it./de/component/customfilters/
? If all your heavy AJAX requests go below that path, but not anything else that is supposed to still be accessible to search engines … then simply say you don’t want anything below that path to be crawled/indexed. And/or you might want to tell the Yandex bot to slow down and keep a certain pause between requests – a quick search suggests that should be easily possible as well.