FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  Htaccess  »  Alternate robots.txt files with Htaccess

by 1 comment

Alternate robots.txt files with HtaccessSo here's the basic idea: There are 2 sites, a development site and a live site. They are essentially mirrors of each other in terms of they have the same files. You need to disallow all search engine robots from indexing and crawling the development site, while allowing full crawling of your live site. Htaccess to the rescue!


Create a robots-off.txt

You already should have a robots.txt file, now you just need to create a robots-off.txt file in the same directory as the robots.txt file. This blocks all legitimate search engines.

User-agent: *
Disallow: /

Htaccess Rewrite for Alternate robots.txt

The below code is simple! It just checks the HTTP_HOST to see if it starts with "development", and if so (development.site.com) it internally rewrites (not redirects) requests for /robots.txt to /robots-off.txt

###
### Alt robots.txt ala askapache.com/htaccess/alternate-robots-txt-rewrite.html
###
RewriteCond %{HTTP_HOST} ^development.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-off.txt [NC,L]

Tags

February 22nd, 2013

Comments Welcome

  • Nikola Marić

    How should I know if I did it right? Is there any way to check if everything is working as it should?

Popular Articles
My Online Tools

Related Articles
Newest Posts
Twitter



Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman






[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

| Google+ | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain