Now just sit back and pray it works

However for human reading purposes you might put your normal mod rewrite rules (SEO urls) on top below the engine and php flags followed by all your blocking rules.

You are right! It didn’t work for me either, but I figured it out. One thing that could be at fault is if you have the SetEnvIF code at the bottom of your .htaccess file, put it at the top. Another Likely reason is if your server is using suexec, which limits Environment variables to safe names.

I updated the example .htaccess code above to show the correct code, including libwww-perl as well.

1. How can/should I test whether-or-not my ‘bad_web_bot’ has been set?
2. Can you suggest a SetIfNoCase User-Agent for empty user agents, and
3. You seem to be the only place that uses the syntax ‘User-Agent’. Shouldn’t it be ‘http_user_agent’. (My experience seems to indicate cases are insensitive. Not so?)

1.

change bad_web_bot to HTTP_SAFE_BADBOT to keep it suexec safe. Then you can test whether it’s been set by using mod_rewrite, mod_headers, mod_setenvif, etc..

2.

SetEnvIf ^User-Agent$ "^$" HTTP_SAFE_EMPTY_UA
deny from env=HTTP_SAFE_EMPTY_UA

3.

SetEnvIf only has 6 variables it can access, and those are specific to mod_setenvif. What SetEnvIf is good at is parsing the HTTP REQUEST HEADERS such as the User-Agent request header. And SetEnvIf is case-insensitive when dealing with headers. HTTP_USER_AGENT is a variable used by mod_rewrite.

SetEnvIfNoCase User-Agent .*(libwww-perl|aesop_com_spiderman).* bad_web_bot

… might not be working.

For example, I’ve still get libwww-perl appearing in my access logs. As a test I tried adding in a portion of text appearing from my own user agent string to see if I could block myself … alas … I was able to still access my website.

Three questions:

1. How can/should I test whether-or-not my ‘bad_web_bot’ has been set?
2. Can you suggest a SetIfNoCase User-Agent for empty user agents, and
3. You seem to be the only place that uses the syntax 'User-Agent'. Shouldn’t it be 'http_user_agent'. (My experience seems to indicate cases are insensitive. Not so?)

Thanks

It will slow down the apache httpd server process, but it won’t be noticeable unless you have a crazy-high traffic site.

The Rewrite Engine (if using RewriteRules to block) looks at the incoming request, performs the rewrites, and then apache serves the response. So by adding more RewriteRules for the RewriteEngine to process, you theoretically add more processing and time to each request being rewritten, but this “extra” time added will almost definately be unnoticed.

@ Tom Dawkings

Yes, please read: 57 HTTP Status Codes and Apache ErrorDocuments

@ Bernie

Nice one, just updated the code..

Actually that line is targeting any user-agent starting with web and containing any of the items in parentheses.. Its a shortcut to typing (webzip|webemaile|webenhancer) so the correct line is:

SetEnvIfNoCase ^User-Agent$ .*web(zip|emaile|enhancer).* bad_web_bot

SetEnvIfNoCase ^User-Agent$ .*(web(zip|emaile|enhancer).* bad_web_bot

Shouldn’t this be

SetEnvIfNoCase ^User-Agent$ .*(webzip|emaile|enhancer).* bad_web_bot

Now, with so many options for dealing with security, at least I’ve got this one running for now. Thanks.