Home » Htaccess » Alternate robots.txt files with Htaccess

by 1 comment

robots-off-txtSo here's the basic idea: There are 2 sites, a development site and a live site. They are essentially mirrors of each other in terms of they have the same files. You need to disallow all search engine robots from indexing and crawling the development site, while allowing full crawling of your live site. Htaccess to the rescue!

Create a robots-off.txt

You already should have a robots.txt file, now you just need to create a robots-off.txt file in the same directory as the robots.txt file. This blocks all legitimate search engines.

User-agent: *
Disallow: /

Htaccess Rewrite for Alternate robots.txt

The below code is simple! It just checks the HTTP_HOST to see if it starts with "development", and if so ( it internally rewrites (not redirects) requests for /robots.txt to /robots-off.txt

### Alt robots.txt ala
RewriteCond %{HTTP_HOST} ^development.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-off.txt [NC,L]


Comments Welcome

AskApache is an FSF Contributing Member

Related Articles

My Online Tools
Popular Articles

Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman


It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain