Comments on: Blocking Bad Bots and Scrapers with .htaccess

By: sunny Tewathia

sunny Tewathia — Thu, 03 Jan 2013 17:00:23 +0000

How can i specifically block a site or any type of bot using by that site to scrap my content. It is scraping my content and publishing as their own.
Please help.

Thanks.

By: Name rod

Name rod — Mon, 12 Nov 2012 21:14:16 +0000

hey great list and methodologies here, it must have taken a lot of work and study to produce this. sorry but am a pure simpleton (but know a little c++). Has anyone ever provided a list of good bots to only allow these in. The list of bad bots must grow exponentially but good bots only linearly? regards rod

By: Ryan

Ryan — Fri, 08 Jun 2012 23:10:06 +0000

For nginx users:

if ($http_user_agent ~*
"^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack)"){
        set $rule_0 1;
        return 403;
        break;
}
if ($http_user_agent ~*
"^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$"){
        set $rule_0 1;
        return 403;
        break;
}

By: Techgazine

Techgazine — Sun, 11 Mar 2012 13:47:23 +0000

Would you know how to block the WP Robot autoblogging plugin? It's scraping my other websites. Thanks

By: Phoenix

Phoenix — Sun, 15 Jan 2012 22:04:20 +0000

is it normal to still have logs of bad bots with RewriteRules?

And another question, what is the best solution: redirection or forbidden ?

RewriteRule ^(.*)$ http://www.example.com/ [R,L]
RewriteRule . - [F,L]

Thanks by advance for your answer and thanks for this great post! :)

By: Anuj@WordPress SEO

Anuj@WordPress SEO — Tue, 15 Nov 2011 07:34:59 +0000

You're post is good... I agree that people should make the most of the built-ins before jumping to the advanced modules. That's precisely what I'm doing, however, I'm having some trouble and was hoping you might help.

By: Erika

Erika — Mon, 03 Jan 2011 17:26:56 +0000

¿En dónde se pega ese código para evitar que se bajen el sitio web?

By: memoi

memoi — Sun, 19 Dec 2010 14:00:10 +0000

thanks for the post.
I am looking for a way to block/deny OneNote. Is there a certain rule to deny OneNote or any program ?

many thanks

By: mark

mark — Sat, 06 Nov 2010 20:58:06 +0000

Very good, thanks for placing this online! There are scripts that set traps for bad bots but somehow I can't set them to block anything... at least with this a few bad bots are trapped :)

By: Donna

Donna — Thu, 18 Mar 2010 19:17:18 +0000

I have 2 questions:

Why do we need a custom 404 page?
Below is my htaccess. To save space I've removed some code. Will this work? Do I need multiple "RewriteEngine On" statements? If not, do I keep the top one and then just begin your code with "Rewrite Base"? Thank you!

# redirect non-www to the www url always
Options +FollowSymLinks
RewriteEngine on
...
# prevent hotlinking of images
RewriteEngine on
....
# custom not found file
ErrorDocument 404 /notfound.shtml
 
# beginning of the blocking of bad bots
ErrorDocument 403 /403.html
RewriteEngine On
RewriteBase /
 # IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
.............rest of rewrites...............
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]