Comments on: WordPress robots.txt SEO

By: MakeMoney

MakeMoney — Tue, 27 Oct 2009 05:35:26 +0000

We try this robots for our blog. Hope get more value for SEO.

By: francis

francis — Wed, 23 Sep 2009 19:48:15 +0000

Great! Thanks for sharing!

By: marco

marco — Tue, 18 Aug 2009 10:00:12 +0000

I know that this article is outdated, but just one question.
As far as I know, wildcards are not supported in robots.txt (it should be based only on substring matches: http://www.robotstxt.org/faq/robotstxt.html), only “User agent:*” is accepted.

So are you sure your robots.txt (even the new one) works as you expect? Are then wildcards supported?

By: Jonathan

Jonathan — Tue, 23 Jun 2009 14:34:36 +0000

I have been doing some reading on SEO. I just read this blog. I am using a robots plug in. Is the robots plug in a complement or a replacement to your script? (I am a newbie. Please take it easy.)

By: Vikram

Vikram — Wed, 06 May 2009 16:30:06 +0000

Thanks, thats what i was looking for, SEO friendly robots.txt

By: AskApache

AskApache — Sun, 08 Mar 2009 09:55:50 +0000

@ira I would recommend much leaner, so you don't block the wrong bot. Remember that robots.txt is taken quite literally/seriously by search engines, better to use meta tags for fine-grain access control. Nice work though!

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /tag/
Disallow: /hakkinda/
Disallow: /iletisimvereklam/
Disallow: /wp-content/upgrade
Disallow: /wp-content/plugins
Disallow: /wp-content/languages
Disallow: /wp-content/themes/default
Disallow: /wp-content/themes/guzel-pro
Disallow: /wp-content/themes/wp-max
Disallow: /wp-content/themes/classic
Disallow: /.smileys
 
User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /z/
Disallow: /wp-*
Allow: /wp-content/uploads/
 
 
# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*
 
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Allow: /*
 
# disallow archiving site
User-agent: ia_archiver
Disallow: /
 
# disable duggmirror
User-agent: duggmirror
Disallow: /

By: ira

ira — Sun, 08 Mar 2009 05:34:24 +0000

Hello Apache i am avid reader of your blog. I made a robots txt file based on the information you gave is it ok is there any mistakes? Thanks.

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /stats/
Disallow: /dh_
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /tag/
Disallow: /hakkinda/
Disallow: /iletisimvereklam/
Disallow: /wp-content/upgrade
Disallow: /wp-content/plugins
Disallow: /wp-content/languages
Disallow: /wp-content/themes/default
Disallow: /wp-content/themes/guzel-pro
Disallow: /wp-content/themes/wp-max
Disallow: /wp-content/themes/classic
Disallow: /trackback/
Disallow: */trackback/
Disallow: /.smileys
 
User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.cgi$
Disallow: /*.wmv$
Disallow: /*.png$
Disallow: /*.gif$
Disallow: /*.jpg$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: */trackback*
Disallow: /z/
Disallow: /wp-*
Allow: /wp-content/uploads/

 
# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow: /*?*
Allow: /z/
Disallow: /hakkinda/
Disallow: /iletisimvereklam/
Allow: /wp-content/
Allow: /tag/
Allow: /manual/*
Allow: /docs/*
Allow: /*.php$
Allow: /*.js$
Allow: /*.inc$
Allow: /*.css$
Allow: /*.gz$
Allow: /*.cgi$
Allow: /*.wmv$
Allow: /*.cgi$
Allow: /*.xhtml$
Allow: /*.php*
Allow: /*.gif$
Allow: /*.jpg$
Allow: /*.png$

# disallow archiving site
User-agent: ia_archiver
Disallow: /

# disable duggmirror
User-agent: duggmirror
Disallow: /

By: AskApache

AskApache — Thu, 05 Mar 2009 15:01:17 +0000

@rozkan dude this article is 2 years old.. this is a blog not an encyclopedia. Regardless of that, If you follow the instructions in this article you should be in really good shape. If you cut-and-paste my old robots.txt, didn't even bother to check my actual one, and skipped reading the other newer articles here on my blog or anywhere else.. well geez man I'm sorry you had a problem but I can't work miracles. I bet you did actually learn something when it all went wrong though, some people like to learn that way, I do it too sometimes. THINK before you copy.. I'm not your robot.

By: rozkan

rozkan — Tue, 03 Mar 2009 21:35:19 +0000

Thanks genius! I used these commands for robots.txt and a week later i realized my new pages weren't indexed by googlebot! By the way if this "ultimate super best robots.txt" why don't YOU use it ha? I see your robots.txt file is pretty simple! So you are lying!

By: GoSimple

GoSimple — Fri, 27 Feb 2009 07:52:50 +0000

Perfect! Cheers.