“Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results.”

If that is not proof that Google does anything that it feels like doing, then what is?
Yeah, right, sure … Google respects your robots.txt directives not to crawl a given webpages. Google is going to return its content in the search results!!!
With double-talk like this, I am going to depend on using my own brain and not read more dribble from Matt Cutts.

In an effort to reduce potential dupe content, applied a robots.txt recently based on one I copied from an article elsewhere in which the author was using disallow for his index.php, not sure what I was thinking, that was one of the lines that ended up in my robots.txt and unless there is some other unknown aspect involved, (no messages from Google…), for the time being I seem to be literally wiped off the face of Google. Hopefully this is just temporary the Google bot revisits and starts indexing from my home page again… My Home Page (index.php) in fact was my highest ranked page and was ranked very attractively in my desired search results – argggh

So- after reading around, including the interview you posted, I started thinking (and have little to lose for the moment) and this is probably the eternal conflict for anyone wishing to SEO their site on anything but the most basic of levels, “how do I get ALL my content seen but avoid dupe content penalties? have decided to try the following – using noindex, follow tags on the content most likely to be flagged for due content (I use a excerpting plugin as it is for anything resembling an archive…) and only using disallow for for administrative or site code areas (i.e. the wp-* folders) and server areas i.e. cgi-bin as they aare surely duplicates of many others sites, its boring stuff that has no business showing up in content search results and just generally feeling like its best to keep as much of the administraive and code stuff private for security reasons.

So my thinking is, dont ask the googlebot to index all the flotsam and jetsam (noindex – archives, other likely “dupy” content…) but tell it to follow so any linky goodness is maximised – i.e outbound links contained in any of the content – this is all seems pretty wholesome?

Or am I just somehow missing the boat in my naivety?

I just wondered to myself if the follow tag needs to be included, its probably default action for a searchbot?

Any way thanks askapache for making some interesting and thought provoking content available on your site.

Nice advice, thanks.

Leak juice? I prefer not to humanize the SEO process. Your post once again brings up the importance of internal link structure. Use NoFollow on your “about us page”? An absolute waste of time, IMHO. Webmasters should use NoFollow to artificially fix an inherit weakness of the Google Page Rank system? No, I have better things to do with my time. Like making my “about us page” into something important and worth reading. And, personally I think webmasters have better things to do than waste their time on time consuming meaningless busy work.

Also, I’m ranked 12th or so for my key phrase and hoping to land on the home page soon. Will this adversely affect my ranking in any way or could it help? Its taken a while to get here so I hate to rock the boat unless it will likely help me.
Thanks