Post by AskApache | Feb 22, 2013
So here's the basic idea: There are 2 sites, a development site and a live site. They are essentially mirrors of each other in terms of they have the same files. You need to disallow all search engine robots from indexing and crawling the development site, while allowing full crawling of your live site. Htaccess to the rescue!
Htaccess Htaccess RewriteRule robots.txt
Mar 15, 2008
Implementing an effective SEO robots.txt file for WordPress will help your blog to rank higher in Search Engines, receive higher paying relevant Ads, and increase your blog traffic. Get a search robots point of view... Sweet!
SEO Google meta robots robots.txt SEO
May 10, 2007
WordPress robots.txt file can make a huge impact on your WordPress blogs traffic and search engine rank. This is an SEO optimized robots.txt file.