15

I would like to block all the http_user_agents which identify as bots but allow Googlebot when I put the following code:

map $http_user_agent $bad_bot {
default 1;
~*^Lynx 0; # Let Lynx go through
~*^google );
libwww-perl                      1;
~(?i)(libwww|Wget|LWP::Simple|BBBike|java|crawl|spider|bot) 1;
}

however, this block access to even googlebot.

2 Answers 2

42

Here my logic for nginx

map $http_user_agent $limit_bots {
     default 0;
     ~*(google|bing|yandex|msnbot) 1;
     ~*(AltaVista|Googlebot|Slurp|BlackWidow|Bot|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker) 1;
     ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|Go!Zilla|Go-Ahead-Got-It|GrabNet|Grafula|HMView|Go!Zilla|Go-Ahead-Got-It) 1;
     ~*(rafula|HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|tool|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE) 1;
     ~*(GrabNet|NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider) 1;
     ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus) 1;
     ~*(Twengabot|htmlparser|libwww|Python|perl|urllib|scan|Curl|email|PycURL|Pyth|PyQ|WebCollector|WebCopy|webcraw) 1;
 } 

location / {
  if ($limit_bots = 1) {
    return 403;
  }
}
4
  • 13
    People should note that this method will disable google from indexing your website via catching the word "google" and the word "bot" (because google's http_user_agent had the word bot in it.
    – bmiskie
    Mar 25, 2016 at 4:20
  • 1
    @bmiskie Good clarification, thank you. In my case, that's exactly what I want to do!
    – emc
    Feb 20, 2018 at 7:10
  • 1
    is there a way to skip those entries being logged ?
    – E Ciotti
    Nov 16, 2018 at 15:25
  • A useful list of bots.
    – Ravi Soni
    Oct 18, 2022 at 6:52
28

Simply check $http_user_agent against your $bad_bot list and return HTTP 403 if it's in your blacklist:

location / {
   if ($http_user_agent ~ (libwww|Wget|LWP|damnBot|BBBike|java|spider|crawl) ) {
       return 403;
   }
}

Note: ~ in the if block performs case-sensitive match. If you want to make your blacklist case-insensitive, use ~* instead of ~.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.