Well its because otherwise a search for askapache on google might list urls like http://www.askapache.com/this-seo-post/great-url.html#comment-232497 which is what it means to have a URL indexed in a search engine.

I have about 230 posts on this blog, all high-quality, and coincidentally I have about 240 urls indexed by Google and major search engines. So it really makes my good pages the center of attention.

I guess it is to prevent specific comment URL from being indexed but the format of the comments URL isn’t like this, isn’t it?

nice blog btw.

I am not handy in Robots.txt file, but would i block whole pages from search engines incase if i use both your sample robots.txt file and php codes that you provided for Wordpress.

But could you please tell more about following strings:

Disallow: (without any symbol after colon)
Allow: /*
Disallow: /

What do they mean? Is there any difference between “Allow: /*” and “Disallow:“?

And how I should disallow indexing of a particular directory: “Disallow: /wp-admin” or “Disallow: /wp-admin/“? (should I use slash at the end or not?)

yo propongo uno como :
http://www.vuelomania.com/robots.txt

@ cosasdeviajes

Why you are not disallowing /2007, /author and /page ???

I fixed it so that now it disallows /2007 and /author and I’m allowing /page in my robots.txt so that bots can still follow the links on /page* but they will not be indexed. This makes sure that they don’t use up any link-juice on my site and also helps search engines find more interlinking between my main content, single pages.

<meta name="robots" content="noindex" />

Sitemap: http://www.xxx.es/sitemap.xml
 
User-agent: *
Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /?s=
Disallow: /dogs
Disallow: /archives
Disallow: /page
Disallow: /author
Disallow: /2007
Disallow: /category
Disallow: /2008
Disallow: /2009
Disallow: /?livehit=

My issue is that my post only ranks when are on the homepage, is something wrong with my robots.txt ??? I´m ussing the same robots.txt on other two blogs and ranks really well

Or I need to add this only on index or default page so that robots can follow all links from there?

Also what will happen if robot lands on a inner page first? does this line helps re-directing robot to follow links from index page?

I hear this is good, but should I then block my original wordpress categories?

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-*

I think I need some change.
Thanks 4 your post:)