Comments on: WordPress robots.txt SEO

By: Tyrone Campbell

Tyrone Campbell — Mon, 04 Aug 2008 17:49:51 +0000

Thats a really good system and plugin since i run a wordpress mu site this will really help out people using the site, one other thing ive been looking for it a sitemap generator that works for wordpress mu, know of any? (email me answer please :D )

thanks again!

By: John Doro

John Doro — Fri, 11 Jul 2008 23:35:09 +0000

I’d like to try it, but how safe is this? Can we undo this and re-gain our PR supposed this customization won’t work?

By: AskApache

AskApache — Thu, 29 May 2008 01:34:51 +0000

@Rene

Its a very confusing subject.. But you can find out what a robots.txt does in general just by reading through a couple posts on this blog.

Basically this post is my best robots.txt at that point in time, and the robots.txt I have now is even better.. for me.

All robots.txt does is tell search engines what to NEVER include in search engine results, so I have moved more towards keeping the robots.txt more open and using meta tags instead.

By: Rene Dwight

Rene Dwight — Thu, 22 May 2008 19:49:02 +0000

This is an amzing post but I have a couple of questions,,

How can adding the above robots.txt file boost rank and add traffic?

Also please forgive me if I am being dumb.. I checked the robots txt file for your site and it is not like the examples above? I don’t quite understand?

Kindest regards Rene

By: max

max — Mon, 24 Mar 2008 16:51:17 +0000

Can you also include the noarchive command for the robots and msnbot lines, as well as the googlebot line? If not, why?

By: eBlogger

eBlogger — Thu, 04 Oct 2007 11:50:42 +0000

I was looking for a detailed Search Engine Optimization article on the robots.txt
This is really detailed and useful.

Thanks

:-)