Comments on: Updated robots.txt for WordPress

By: Robertino

Robertino — Sat, 14 Nov 2009 09:05:47 +0000

My previous posts are from October 1st/2nd 2009.
You shouldn’t worry about, this information is very static.
It’s not like they are changing the WordPress directory names, nor what’s in them frequently.
If it would change frequently, the theme designers, plugin coders, blog owners, and search engines wouldn’t bother.
What’s most likely to change (frequently) are your own directory names and their content.
If you don’t update that part of the robots.txt file, then it will be old information.

Off Topic 1:
Tip : Be careful with robots.txt. It’s a simple text file, that anyone can show in their browser.
Don’t giveaway your whole site structure, use meaningless names for the directories that you wish to protect.

Off Topic 2:
Bloody Hell, I see that Mehmet of GabfireThemes posted in here. How could I have missed that before ?

By: Dick Raney

Dick Raney — Fri, 13 Nov 2009 22:48:03 +0000

How old is this information? Why don’t you show dates on the post or the comments?

By: hpnugroho

hpnugroho — Mon, 12 Oct 2009 14:46:56 +0000

hi ..
I have question about this file, can we use on free blog on wordpress.com or blogspot.com ?
Thanks for your information

By: Ujjwol

Ujjwol — Fri, 02 Oct 2009 04:44:57 +0000

Thanks Guys

By: Robertino

Robertino — Fri, 02 Oct 2009 03:13:32 +0000

@Ujjwol If you really want archive.org to duplicate your content remove the following.

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: ia_archiver
Disallow: /

But archive.org is a reasonably good way to proof that your site's content existed at a certain date and was not parked with a an all ads page, or had the content first (in case of plagiarism).

By: Ujjwol

Ujjwol — Thu, 01 Oct 2009 05:14:55 +0000

Archive.org says it cannot crawl my website due to this robots.txt ?
How to fix this ?

By: Robertino

Robertino — Tue, 22 Sep 2009 08:03:26 +0000

Forgot to add the following from robotstxt.org. I'm going with the format of your robots.txt file for now. ( http://www.robotstxt.org/robotstxt.html )

Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".

By: Robertino

Robertino — Tue, 22 Sep 2009 07:06:36 +0000

I'm setting up a new blog and this time wanted to use a proper robots.txt file. So I started reasearching. Over at wordpress.org, where this page is linked from ( just above the grey box : Search Engine Optimization for WordPress ) I see that wildcards are used in the robots.txt file sample. Then I clicked the link and landed here, and this robots.txt file is not using wildcards at all. This, and the examples over at robotstxt.org, are the first and only robots.txt file for WordPress which I have seen that are not using the asterisk * sign as a wildcard. This is I *think* a good thing, because I read over at : robotstxt.org the following :

Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.

Two common errors:

Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp/'.
You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec)

But still I see countless of other sites using wildcards in paths. Was this changed so that wildcards are supported, and maybe the robotstxt.org site just not updated? One other thing, I see the lines 'Disallow: /i/' etc in your robots.txt file. Do these have to do with the fact that typing the first letter of a page take you to that page ? For example : domain.com/c/ and domain.com/c goes to the contact page. When I enter for instance domain.com/i/ in my browser's address bar, I get a "No posts found" message with a "Nothing found for I" in the title. Should I stll be adding those lines ? Or should I just go for the whole alphabet ? :) Apologies for the lengthy post.

By: Jon @ IBM Engineering

Jon @ IBM Engineering — Fri, 14 Aug 2009 22:56:29 +0000

thanks :) It seems I was really thinking to much into robots.txt files, they are really just a simple old school method to block search engine crawlers in a very simple way. I was all confused with wordpress robots .txt files but they really are that simple.

Thanks and have a great day!

Jon @ IBM Core

By: Mr. T

Mr. T — Sun, 02 Aug 2009 09:52:40 +0000

Excellent post. I shall be using your robots.txt on my site and hopefully I’ll see good results enough to make write a short post on this and link to your site. Thanks a lot.