FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  Htaccess  »  Alternate robots.txt files with Htaccess

by comment

Alternate robots.txt files with HtaccessSo here's the basic idea: There are 2 sites, a development site and a live site. They are essentially mirrors of each other in terms of they have the same files. You need to disallow all search engine robots from indexing and crawling the development site, while allowing full crawling of your live site. Htaccess to the rescue!


Create a robots-off.txt

You already should have a robots.txt file, now you just need to create a robots-off.txt file in the same directory as the robots.txt file. This blocks all legitimate search engines.

User-agent: *
Disallow: /

Htaccess Rewrite for Alternate robots.txt

The below code is simple! It just checks the HTTP_HOST to see if it starts with "development", and if so (development.site.com) it internally rewrites (not redirects) requests for /robots.txt to /robots-off.txt

###
### Alt robots.txt ala askapache.com/htaccess/alternate-robots-txt-rewrite.html
###
RewriteCond %{HTTP_HOST} ^development.*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*robots\.txt.*\ HTTP/ [NC]
RewriteRule ^robots\.txt /robots-off.txt [NC,L]

Tags

February 22nd, 2013

Comments Welcome

My Online Tools
WordPress Sites

My Picks

Related Articles
Newest Posts
Twitter

  • ZERO DAY - read before Trojan horse  t.co/pPMLGDJv8P 
  • Trojan Horse, a novel!  t.co/Hf8EtYaZVa 
  • The Hacker Playbook - very nice high level overview of attacks  t.co/lHwNVWi61u 
  • Clean Code - A Handbook of Agile Software Craftsmanship  t.co/hnJX0x1qIc 
  • Secrets of the JavaScript Ninja - By my absolute favorite JS hacker John Resig!  t.co/tZ42ljmcCl 
  • Hacking Exposed 7: Network Security Secrets & SolutionsMy all time favorite, basic but thorough and accurate.  t.co/jycW0RDVtZ 
  • Empty words will be no surrogate for cold resolve. Pain is nothing.  t.co/qXjpRxbjCw 
  • REVERSING: Secrets of Reverse Engineering  t.co/GaWo29lWWG 
  • NEUROMANCER  t.co/3OoknUcb5Z 
  • "The Shockwave Rider", by John Brunner (1975 hacker sci-fi)  t.co/ZW56HVUefW 
  • The Rootkit ARSENAL - Escape and Evasion in the Dark Corners of the System  t.co/1FzX6bHgsQ 
  • "We Are Anonymous - Inside the Hacker World of LulzSec, Anonymous, and the Global Cyber Insurgency" better be good!  t.co/GL0cFNiUOq 
  • THE IDEA FACTORY Bell Labs  t.co/FyVhgNwwT5 
  • The Datacenter as a Computer -- Urs Holzle  t.co/M5WIYs1OVg 

Friends and Recommends
Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman






[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

| Google+ | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain