Updated robots.txt for WordPress

Contents

1. WordPress robots.txt SEO
   1. static.askapache.com/robots.txt
2. Google Recommendations
   1. Use robots.txt - Webmaster Guidelines
   2. Troubleshooting tips part IIb: Ad relevance and targeting continued
   3. Eliminate Duplicate Content
   4. Prevent page from being indexed
   5. Prevent content being indexed or remove content from Google's index?
   6. Google User-agents
3. Good Robots.txt Articles
4. Robots Meta Tags
   1. Robots Meta Examples
      1. Stop all robots from indexing a page on your site, but still follow the links on the page
      2. Allow other robots to index the page on your site, preventing only Googles bots from indexing the page
      3. Allow robots to index the page on your site but not to follow outgoing links
   2. header.php Trick for Conditional Robots Meta
      1. Add this to your header.php
   3. Robots.txt References

• Feed for this Entry
• Trackback
• redirect htaccess

Reader Comments

1. Pingback:SEO Secrets of AskApache Part 2

2. sSysAdmin ~October 9, 2010 @ 10:38 pm

   Amazing post, I'm newbie in blogging and this article helped me a lot to understant robots.
3. Spunky Jones ~May 29, 2010 @ 3:23 am

   I am using the following in my robots.txt file on one of my directories. I noticed that I have over 1,700 pages and only 400 pages indexed by Google. My question is, which statements in my robots.txt file are causing the pages not to get index? Furthermore, do I actually need any of this in the robots.txt file?

   #  PARTIAL access (Googlebot)
   User-agent: Googlebot
   Disallow: /*.php$
   Disallow: /*?
   Disallow: /*?*
   Disallow: /?s=A&p=1
   Disallow: /*/trackback/

4. Alan ~May 2, 2010 @ 9:09 pm

   Hi again, I commented a few days back but it didn't seem to take so here goes again... :-) I understand that WP has a virtual robots.txt file. The xml sitemap plugin has an option to add the Sitemap to the virtual robots file. I have chosen that option and the virtual robots.txt file is recognized by Google Webmaster Tools and does in fact include the Sitemap. The sitemap plugin states the following: The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory! So, what to do? Is the correct answer to not choose the option via the xml sitemap plugin to use the virtual robots.txt file and to create a new actual robots.txt file, where I can also include the Sitemap? Also, why include the Allow option in the robots.txt file for individual Google bots? Why not just Allow all (*) ? And why use any "Allow" command at all? Google will crawl the site providing that there is no Disallow command. I look forward to your reply. Thanks!
5. zedcloud ~April 4, 2010 @ 11:22 pm

   Nice article! What's the recommended file permissions for robots.txt? Is 644 fine? For some reason my site is being detected as a directory and says specific files may have different restrictions. As a result site is not indexed in Google. Any help?
6. gina ~March 30, 2010 @ 4:26 pm

   Hello, I am not very good in techniques but tired to follow all the way down your recommendations. Today big suprise: I can not enter any more in my blog: The message is as follow;

   Warning: require_once(admin.php) [function.require-once]: failed to open stream: No such file or directory in /public_html/wp-admin/index.php on line 10
   Fatal error: require_once() [function.require]: Failed opening required 'admin.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /public_html/wp-admin/index.php on line 10

   Do you know what this means and can you get me a good advise? Lucky that I made a backup before but do not know now how to restore, because I cannot enter my blog. Also it is not possibly to enter in filezilla. Please let me know what do do!! Thank you Gina
7. Social Media ~March 30, 2010 @ 6:04 am

   Excellent article, thank you. I am not sure what I'm doing with robots files but know that I need one so this is very useful. Thanks again!

8. Pingback:Redirect index.php to root » Server~Techs

9. BM ~March 9, 2010 @ 6:07 am

   thanks, your site has tons of good info, we just finished a post on setting up robots.txt in wordpress to mimic silos you can check it out at Ultimate WordPress Robots.txt for Silo SEO
10. Robertino ~November 29, 2009 @ 2:32 pm

    Off-topic @Dick Raney: Better late then never. Forgot to add that when you hover your mouse pointer over a comment, it will change into an arrow with a question mark beside it. One second later or so, the date and time the comment was posted will also popup.
11. Keith Davis ~November 26, 2009 @ 1:14 pm

    Wow! Pretty comprehensive robots.txt. I'm familiar with not producing duplicate content on a static site but the WordPress duplicate content is something that I'm trying to get to grips with. With archives, posts, categories etc. etc. it's not easy. Thanks for shedding a little light.
12. Robertino ~November 14, 2009 @ 4:05 am

    My previous posts are from October 1st/2nd 2009. You shouldn't worry about, this information is very static. It's not like they are changing the WordPress directory names, nor what's in them frequently. If it would change frequently, the theme designers, plugin coders, blog owners, and search engines wouldn't bother. What's most likely to change (frequently) are your own directory names and their content. If you don't update that part of the robots.txt file, then it will be old information. Off Topic 1: Tip : Be careful with robots.txt. It's a simple text file, that anyone can show in their browser. Don't giveaway your whole site structure, use meaningless names for the directories that you wish to protect. Off Topic 2: Bloody Hell, I see that Mehmet of GabfireThemes posted in here. How could I have missed that before ?
13. Dick Raney ~November 13, 2009 @ 5:48 pm

    How old is this information? Why don't you show dates on the post or the comments?
14. hpnugroho ~October 12, 2009 @ 10:46 am

    hi .. I have question about this file, can we use on free blog on wordpress.com or blogspot.com ? Thanks for your information
15. Ujjwol ~October 2, 2009 @ 12:44 am

    Thanks Guys
16. Robertino ~October 1, 2009 @ 11:13 pm

    @Ujjwol If you really want archive.org to duplicate your content remove the following.

    User-agent: ia_archiver-web.archive.org
    Disallow: /

    User-agent: ia_archiver
    Disallow: /

    But archive.org is a reasonably good way to proof that your site's content existed at a certain date and was not parked with a an all ads page, or had the content first (in case of plagiarism).
17. Ujjwol ~October 1, 2009 @ 1:14 am

    Archive.org says it cannot crawl my website due to this robots.txt ? How to fix this ?
18. Robertino ~September 22, 2009 @ 4:03 am

    Forgot to add the following from robotstxt.org. I'm going with the format of your robots.txt file for now. ( http://www.robotstxt.org/robotstxt.html )

    Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".

19. Robertino ~September 22, 2009 @ 3:06 am

    I'm setting up a new blog and this time wanted to use a proper robots.txt file. So I started reasearching. Over at wordpress.org, where this page is linked from ( just above the grey box : Search Engine Optimization for WordPress ) I see that wildcards are used in the robots.txt file sample. Then I clicked the link and landed here, and this robots.txt file is not using wildcards at all. This, and the examples over at robotstxt.org, are the first and only robots.txt file for WordPress which I have seen that are not using the asterisk * sign as a wildcard. This is I *think* a good thing, because I read over at : robotstxt.org the following :

    Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.

    Two common errors:

    1. Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp/'.
    2. You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec)

    But still I see countless of other sites using wildcards in paths. Was this changed so that wildcards are supported, and maybe the robotstxt.org site just not updated? One other thing, I see the lines 'Disallow: /i/' etc in your robots.txt file. Do these have to do with the fact that typing the first letter of a page take you to that page ? For example : domain.com/c/ and domain.com/c goes to the contact page. When I enter for instance domain.com/i/ in my browser's address bar, I get a "No posts found" message with a "Nothing found for I" in the title. Should I stll be adding those lines ? Or should I just go for the whole alphabet ? :) Apologies for the lengthy post.
20. Jon @ IBM Engineering ~August 14, 2009 @ 6:56 pm

    thanks :) It seems I was really thinking to much into robots.txt files, they are really just a simple old school method to block search engine crawlers in a very simple way. I was all confused with wordpress robots .txt files but they really are that simple. Thanks and have a great day! Jon @ IBM Core
21. Mr. T ~August 2, 2009 @ 5:52 am

    Excellent post. I shall be using your robots.txt on my site and hopefully I'll see good results enough to make write a short post on this and link to your site. Thanks a lot.
22. Peter ~June 16, 2009 @ 5:21 pm

    Can I mention the robots.txt WordPress plugin? The default content is not the same as yours, but it's certainly a handy way of creating and managing a robots.txt file for WordPress. Official page is at http://wordpress.org/extend/plugins/pc-robotstxt/ Thanks, Peter.
23. GearModa ~April 30, 2009 @ 8:39 pm

    Great tip, actually clarified some questions I had about the robots txt prior. Thanks
24. Prox ~April 18, 2009 @ 3:02 am

    Thank you for this post mate. I got my site indexed! :)
25. pututik ~April 2, 2009 @ 4:54 pm

    Thanks for the good points, my first robots get the SEO friendly but after some accident the file was lost. It will help me to optimize my WP.
26. Bohdan ~March 1, 2009 @ 3:46 pm

    Спасибо за статью, советы и рекомендации. Очень Вам благодарен.
27. Consultor ~February 11, 2009 @ 2:25 pm

    Fantastic article. This article has been of great help to me. good luck in your project
28. JIm Chenoweth ~December 4, 2008 @ 9:40 am

    Ouch, I wish I had seen webdiggers post, thanks for getting 99 percent of my sites content 'resrticted by robot.txt'

    Disallow: /*?*
    Disallow: /*?

    is the same as "get lost robots"

    maybe you should do use all a favor and put disallow in your robot.txt file so this kind of disinformation get weeded out and sifts to the bottom of the sludge pile where it belongs

29. AskApache ~November 21, 2008 @ 3:29 pm

    @ engfer

    Never heard of that, let me search google and see if mine turns up..

    Yep! Check the results of my search site:www.askapache.com Disallow: User-agent: *.

    You can Disallow this by adding this to your robots.txt if you want it removed, I am personally going to keep mine in the index as people use my robots.txt as an example.

    Disallow: /robots.txt

30. engfer ~November 19, 2008 @ 3:29 pm

    Have you seen a problem with Google showing your robots.txt in it's search results?
31. RocyHua ~October 26, 2008 @ 10:20 pm

    Thanks a lot, This robots.txt tutorial is Useful!
32. Renegado ~October 4, 2008 @ 10:53 pm

    Pienso que los comentarios no tienen por qué estar indexados. En todo caso, los comentarios forman parte de cada post. La instrucción

    Disallow: /wp-

    se excede en alcance, ya que dentro de esa carpeta se halla la carpeta /uploads, así que hay que estar seguro de lo que se está haciendo al momento de usarla. Por último le estaremos impidiendo el acceso a googleblogs search si colocamos la orden

    Disallow: /feed/$

    . WARNING...!
33. Emil ~August 30, 2008 @ 10:58 pm

    Well, I think that the above is well done, duplicated content can indeed hurt any website. Good, high-quality content is what Google is looking for, not otherway around. If you are unsure if what askapache.com is trying to establish, simply take a closer look at their #1 PageRank, #2 SERP and you'll get an immediate answer if this is good or not. Thanks, Emil SEO Agent
34. webdiggr ~June 7, 2008 @ 1:45 pm

    Your Robots.txt will block practically the whole site. Robots.txt does not take variables into consideration, so when you do a:

    Disallow: /*?*
    Disallow: /*?

    Its the same as :

    Disallow: /

    Which blocks everything. I suggest you visit http://www.robotstxt.org/faq/robotstxt.html it will explain how wildcards are not supported.
35. AskApache ~April 20, 2008 @ 5:47 pm

    @ Olivier

    Well its because otherwise a search for askapache on google might list urls like www.askapache.com/this-seo-post/great-url.html#comment-232497 which is what it means to have a URL indexed in a search engine.

    I have about 230 posts on this blog, all high-quality, and coincidentally I have about 240 urls indexed by Google and major search engines. So it really makes my good pages the center of attention

    .
36. Olivier ~February 17, 2008 @ 12:23 pm

    May I please ask you why you put this line : Disallow: */comments I guess it is to prevent specific comment URL from being indexed but the format of the comments URL isn't like this, isn't it?
37. Will ~February 8, 2008 @ 5:21 pm

    I was wondering why the robots.txt file in this example is different to the one at askapache.com/robots.txt nice blog btw.
38. AskApache ~January 21, 2008 @ 5:58 pm

    @ mehmet

    I'm not sure what you are asking, but all the information you need is on this page.

39. Mehmet ~January 17, 2008 @ 11:32 am

    Hi Matt, I am not handy in Robots.txt file, but would i block whole pages from search engines incase if i use both your sample robots.txt file and php codes that you provided for WordPress.
40. David ~January 9, 2008 @ 4:40 pm

    Thanks for the post, it's really useful! But could you please tell more about following strings:

    Disallow: (without any symbol after colon)
    Allow: /*
    Disallow: /

    What do they mean? Is there any difference between "Allow: /*" and "Disallow:"? And how I should disallow indexing of a particular directory: "Disallow: /wp-admin" or "Disallow: /wp-admin/"? (should I use slash at the end or not?)
41. vuelomania – vuelos baratos ~November 12, 2007 @ 5:38 pm

    Hay que limitar aceso a la carpetas para lascuales nos interesa limitar el rastreo, sin embargo cuidado en no occurir en el Blackhat, algunos manipulan los CSS, y limitan el aceso al buscador para que no se de cuenta de la adaptacion de los H1...Hx... yo propongo uno como : http://www.vuelomania.com/robots.txt
42. AskApache ~November 1, 2007 @ 11:00 pm

    @ stacey

    Great!

    @ cosasdeviajes

    Why you are not disallowing /2007, /author and /page ???

    I fixed it so that now it disallows /2007 and /author and I'm allowing /page in my robots.txt so that bots can still follow the links on /page* but they will not be indexed. This makes sure that they don't use up any link-juice on my site and also helps search engines find more interlinking between my main content, single pages.

    <meta name="robots" content="noindex" />

43. stacey ~October 24, 2007 @ 10:13 pm

    I fixed it. I put it in my root directory!
44. cosasdeviajes ~September 17, 2007 @ 12:33 pm

    Why you are not disallowing /2007, /author and /page ???
45. cosasdeviajes ~September 17, 2007 @ 12:12 pm

    Hello, my robots.txt is the following

    Sitemap: http://www.xxx.es/sitemap.xml
     
    User-agent: *
    Disallow: /wp-
    Disallow: /search
    Disallow: /feed
    Disallow: /comments/feed
    Disallow: /feed/$
    Disallow: /*/feed/$
    Disallow: /*/feed/rss/$
    Disallow: /*/trackback/$
    Disallow: /*/*/feed/$
    Disallow: /*/*/feed/rss/$
    Disallow: /*/*/trackback/$
    Disallow: /*/*/*/feed/$
    Disallow: /*/*/*/feed/rss/$
    Disallow: /*/*/*/trackback/$
    Disallow: /?s=
    Disallow: /dogs
    Disallow: /archives
    Disallow: /page
    Disallow: /author
    Disallow: /2007
    Disallow: /category
    Disallow: /2008
    Disallow: /2009
    Disallow: /?livehit=

    My issue is that my post only ranks when are on the homepage, is something wrong with my robots.txt ??? I´m ussing the same robots.txt on other two blogs and ranks really well
46. MrGroove ~September 15, 2007 @ 3:05 am

    Nice writeup. Thnx
47. Steven Wong ~August 23, 2007 @ 12:17 am

    Thanks for the great sharing, I am studying it now and plan to implement to all my blog sites.
48. peter ~August 20, 2007 @ 9:41 pm

    great list, thanks!
49. Sahi ~August 16, 2007 @ 5:45 pm

    If I have 10 pages website, Do we need to add Robots index,follow on each page? Or I need to add this only on index or default page so that robots can follow all links from there? Also what will happen if robot lands on a inner page first? does this line helps re-directing robot to follow links from index page?
50. sam ~August 16, 2007 @ 12:19 pm

    is it good to have the tag cloud crawled. since it is a way of humanly categorizing content? I hear this is good, but should I then block my original wordpress categories?
51. Cherife ~August 12, 2007 @ 8:21 pm

    I usually use these codes: User-agent: * Disallow: /cgi-bin Disallow: /wp-* I think I need some change. Thanks 4 your post:)

Add Comment!