« Hack WP-Cache for Maximum SpeedIP Abuse Detection for DreamHost »
Updated robots.txt for WordPress
March 15th, 2008
Contents
Implementing an effective SEO robots.txt file for WordPress will help your blog to rank higher in Search Engines, receive higher paying relevant Ads, and increase your blog traffic. Using a robots.txt file gives you a search engine robots point of view... Sweet! Looking for the most updated robots.txt? Just look at mine, I don't slack.
WordPress robots.txt SEO
Here are some robots.txt files used with WordPress on this blog. For instance, I am disallowing /category/ in the robots.txt file below because askapache.com/category/htaccess/ is the same as askapache.com/htaccess/, and that would be duplicate content. Adding a 301 Redirect using mod_rewrite or RedirectMatch can further protect myself from this duplicate content issue.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content Disallow: /tag Disallow: /author Disallow: /wget/ Disallow: /httpd/ Disallow: /i/ Disallow: /f/ Disallow: /t/ Disallow: /c/ Disallow: /j/ User-agent: Mediapartners-Google Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: ia_archiver-web.archive.org Disallow: / Sitemap: http://www.askapache.com/sitemap.xml # __ __ # ____ ______/ /______ _____ ____ ______/ /_ ___ # / __ `/ ___/ //_/ __ `/ __ \/ __ `/ ___/ __ \/ _ \ # / /_/ (__ ) ,< / /_/ / /_/ / /_/ / /__/ / / / __/ # \__,_/____/_/|_|\__,_/ .___/\__,_/\___/_/ /_/\___/ # /_/ #
static.askapache.com/robots.txt
User-agent: * Disallow: Allow: /* User-agent: ia_archiver Disallow: / User-agent: duggmirror Disallow: /
Google Recommendations
Use robots.txt - Webmaster Guidelines
Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.
Troubleshooting tips part IIb: Ad relevance and targeting continued
Have you blocked the AdSense crawler's access to your pages?
The AdSense crawler is an automated program that scans your web pages and tracks content for indexing. Sometimes we don't crawl pages because the AdSense crawler doesn't have access to your pages, in which case we're unable to determine their content and show relevant ads. Here are a few specific instances when our crawler can't access a site: If you use a robots.txt file which regulates the crawler access to your page. In this case, you can grant the AdSense crawler access by adding these lines to the top of your robots.txt file:
User-agent: Mediapartners-Google* Disallow:
Eliminate Duplicate Content
Duplicate content generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar. Mostly, this is not deceptive in origin. Examples of non-malicious duplicate content could include:
- Discussion forums that can generate both regular and stripped-down pages targeted at mobile devices
- Store items shown or linked via multiple distinct URLs
- Printer-only versions of web pages
However, in some cases, content is deliberately duplicated across domains in an attempt to manipulate search engine rankings or win more traffic. Deceptive practices like this can result in a poor user experience, when a visitor sees substantially the same content repeated within a set of search results.
Google tries hard to index and show pages with distinct information. This filtering means, for instance, that if your site has a "regular" and "printer" version of each article, and neither of these is blocked in robots.txt or with a noindex meta tag, we'll choose one of them to list. In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we'll also make appropriate adjustments in the indexing and ranking of the sites involved. As a result, the ranking of the site may suffer, or the site might be removed entirely from the Google index, in which case it will no longer appear in search results.
Prevent page from being indexed
Pages you block in this way may still be added to the Google index if other sites link to them. As a result, the URL of the page and, potentially, other publicly available information can appear in Google search results. However, no content from your pages will be crawled, indexed, or displayed.
To entirely prevent a page from being added to the Google index even if other sites link to it, use a noindex meta tag, and ensure that the page does not appear in robots.txt. When Googlebot crawls the page, it will recognize the noindex meta tag and drop the URL from the index.
Prevent content being indexed or remove content from Google's index?
You can instruct us not to include content from your site in our index or to remove content from your site that is currently in our index in the following ways:
- Remove your entire website or part of your website using a robots.txt file.
- Remove individual pages of your website using a robots meta tag.
- Remove cached copies of your pages using a robots meta tag.
- Remove snippets that appear below your page's title in our search results and describe the content of your page.
- Remove outdated pages by returning the proper server response.
- Remove images from Google Image Search using a robots.txt file.
- Remove blog entries from Google Blog Search.
- Remove a feed from our user-agent Feedfetcher, which provides content to our feed readers.
- Remove transcoded versions of your pages (pages we've reformatted for mobile browsers).
Google User-agents
- Adsbot-Google
- crawls pages to measure AdWords landing page quality
- Googlebot
- crawl pages from googles web and news index
- Googlebot-Image
- crawls pages for the image index
- Googlebot-Mobile
- crawls pages for the mobile index
- Mediapartners-Google
- crawls pages to determine AdSense content
Good Robots.txt Articles
- How Google Crawls My Site
- Using the robots.txt analysis tool
- Controlling how search engines access and index your website
- Controlling Access with robots.txt
- Removing duplicate search engine content using robots.txt - Mark Wilson
- Revisiting robots.txt - Twenty Steps
Robots Meta Tags
Robots Meta Examples
Stop all robots from indexing a page on your site, but still follow the links on the page
<meta name="robots" content="noindex,follow" />
Allow other robots to index the page on your site, preventing only Googles bots from indexing the page
<meta name="googlebot" content="noindex,follow" />
Allow robots to index the page on your site but not to follow outgoing links
<meta name="robots" content="nofollow" />
header.php Trick for Conditional Robots Meta
Add this to your header.php
<?php if(is_single() || is_page() || is_category() || is_home()) { ?>
<meta name="robots" content="all,noodp" />
<?php } ?>
<?php if(is_archive()) { ?>
<meta name="robots" content="noarchive,noodp" />
<?php } ?>
<?php if(is_search() || is_404()) { ?>
<meta name="robots" content="noindex,noarchive" />
<?php } ?>
Robots.txt footnote Alexa, Compete, and Quantcast are all guilty of firewalling unknown friendly search engine agents at the front gate. These sites that monitor the Internet should be the most in the know that unfriendly agents cloak as humans and will come in no matter what. So the general rule of thumb is that robots.txt directives are only for the good agents anyway.
Robots.txt References
- Robots.txt optimization
- The Web Robots Pages
- W3.org - Notes on helping search engines index your Web site
- Wikipedia robots.txt page
- Inside Google Sitemaps: Using a robots.txt file
Reader Comments
Pingback:SEO Secrets of AskApache Part 2
-
Amazing post, I'm newbie in blogging and this article helped me a lot to understant robots.
-
I am using the following in my
robots.txtfile on one of my directories. I noticed that I have over 1,700 pages and only 400 pages indexed by Google. My question is, which statements in myrobots.txtfile are causing the pages not to get index? Furthermore, do I actually need any of this in the robots.txt file?# PARTIAL access (Googlebot) User-agent: Googlebot Disallow: /*.php$ Disallow: /*? Disallow: /*?* Disallow: /?s=A&amp;p=1 Disallow: /*/trackback/
-
Hi again, I commented a few days back but it didn't seem to take so here goes again... :-) I understand that WP has a virtual
robots.txtfile. The xml sitemap plugin has an option to add the Sitemap to the virtual robots file. I have chosen that option and the virtual robots.txt file is recognized by Google Webmaster Tools and does in fact include the Sitemap. The sitemap plugin states the following: The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory! So, what to do? Is the correct answer to not choose the option via the xml sitemap plugin to use the virtual robots.txt file and to create a new actual robots.txt file, where I can also include the Sitemap? Also, why include the Allow option in the robots.txt file for individual Google bots? Why not justAllow all (*)? And why use any "Allow" command at all? Google will crawl the site providing that there is no Disallow command. I look forward to your reply. Thanks! -
Nice article! What's the recommended file permissions for robots.txt? Is 644 fine? For some reason my site is being detected as a directory and says specific files may have different restrictions. As a result site is not indexed in Google. Any help?
-
Hello, I am not very good in techniques but tired to follow all the way down your recommendations. Today big suprise: I can not enter any more in my blog: The message is as follow;
Warning: require_once(admin.php) [function.require-once]: failed to open stream: No such file or directory in /public_html/wp-admin/index.php on line 10 Fatal error: require_once() [function.require]: Failed opening required 'admin.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /public_html/wp-admin/index.php on line 10
Do you know what this means and can you get me a good advise? Lucky that I made a backup before but do not know now how to restore, because I cannot enter my blog. Also it is not possibly to enter in filezilla. Please let me know what do do!! Thank you Gina -
Excellent article, thank you. I am not sure what I'm doing with robots files but know that I need one so this is very useful. Thanks again!
-
thanks, your site has tons of good info, we just finished a post on setting up robots.txt in wordpress to mimic silos you can check it out at Ultimate WordPress Robots.txt for Silo SEO
-
Off-topic @Dick Raney: Better late then never. Forgot to add that when you hover your mouse pointer over a comment, it will change into an arrow with a question mark beside it. One second later or so, the date and time the comment was posted will also popup.
-
Wow! Pretty comprehensive
robots.txt. I'm familiar with not producing duplicate content on a static site but the WordPress duplicate content is something that I'm trying to get to grips with. With archives, posts, categories etc. etc. it's not easy. Thanks for shedding a little light. -
My previous posts are from October 1st/2nd 2009. You shouldn't worry about, this information is very static. It's not like they are changing the WordPress directory names, nor what's in them frequently. If it would change frequently, the theme designers, plugin coders, blog owners, and search engines wouldn't bother. What's most likely to change (frequently) are your own directory names and their content. If you don't update that part of the robots.txt file, then it will be old information. Off Topic 1: Tip : Be careful with robots.txt. It's a simple text file, that anyone can show in their browser. Don't giveaway your whole site structure, use meaningless names for the directories that you wish to protect. Off Topic 2: Bloody Hell, I see that Mehmet of GabfireThemes posted in here. How could I have missed that before ?
-
How old is this information? Why don't you show dates on the post or the comments?
-
hi .. I have question about this file, can we use on free blog on wordpress.com or blogspot.com ? Thanks for your information
-
Thanks Guys
-
@Ujjwol If you really want archive.org to duplicate your content remove the following.
User-agent: ia_archiver-web.archive.org Disallow: /
But archive.org is a reasonably good way to proof that your site's content existed at a certain date and was not parked with a an all ads page, or had the content first (in case of plagiarism).User-agent: ia_archiver Disallow: /
-
Archive.org says it cannot crawl my website due to this
robots.txt? How to fix this ? -
Forgot to add the following from robotstxt.org. I'm going with the format of your robots.txt file for now. ( http://www.robotstxt.org/robotstxt.html )
Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".
-
I'm setting up a new blog and this time wanted to use a proper robots.txt file. So I started reasearching. Over at wordpress.org, where this page is linked from ( just above the grey box : Search Engine Optimization for WordPress ) I see that wildcards are used in the robots.txt file sample. Then I clicked the link and landed here, and this robots.txt file is not using wildcards at all. This, and the examples over at robotstxt.org, are the first and only robots.txt file for WordPress which I have seen that are not using the asterisk * sign as a wildcard. This is I *think* a good thing, because I read over at : robotstxt.org the following :
But still I see countless of other sites using wildcards in paths. Was this changed so that wildcards are supported, and maybe the robotstxt.org site just not updated? One other thing, I see the lines 'Disallow: /i/' etc in your robots.txt file. Do these have to do with the fact that typing the first letter of a page take you to that page ? For example :Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.
Two common errors:
- Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp/'.
- You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec)
domain.com/c/anddomain.com/cgoes to the contact page. When I enter for instancedomain.com/i/in my browser's address bar, I get a "No posts found" message with a "Nothing found for I" in the title. Should I stll be adding those lines ? Or should I just go for the whole alphabet ? :) Apologies for the lengthy post. -
thanks :) It seems I was really thinking to much into robots.txt files, they are really just a simple old school method to block search engine crawlers in a very simple way. I was all confused with wordpress robots .txt files but they really are that simple. Thanks and have a great day! Jon @ IBM Core
-
Excellent post. I shall be using your robots.txt on my site and hopefully I'll see good results enough to make write a short post on this and link to your site. Thanks a lot.
-
Can I mention the robots.txt WordPress plugin? The default content is not the same as yours, but it's certainly a handy way of creating and managing a robots.txt file for WordPress. Official page is at http://wordpress.org/extend/plugins/pc-robotstxt/ Thanks, Peter.
-
Great tip, actually clarified some questions I had about the robots txt prior. Thanks
-
Thank you for this post mate. I got my site indexed! :)
-
Thanks for the good points, my first robots get the SEO friendly but after some accident the file was lost. It will help me to optimize my WP.
-
Спасибо за статью, советы и рекомендации. Очень Вам благодарен.
-
Fantastic article. This article has been of great help to me. good luck in your project
-
Ouch, I wish I had seen webdiggers post, thanks for getting 99 percent of my sites content 'resrticted by robot.txt'
Disallow: /*?* Disallow: /*?
is the same as "get lost robots"
maybe you should do use all a favor and put disallow in your robot.txt file so this kind of disinformation get weeded out and sifts to the bottom of the sludge pile where it belongs
-
Have you seen a problem with Google showing your robots.txt in it's search results?
-
Thanks a lot, This
robots.txttutorial is Useful! -
Pienso que los comentarios no tienen por qué estar indexados. En todo caso, los comentarios forman parte de cada post. La instrucción
Disallow: /wp-
se excede en alcance, ya que dentro de esa carpeta se halla la carpeta /uploads, así que hay que estar seguro de lo que se está haciendo al momento de usarla. Por último le estaremos impidiendo el acceso a googleblogs search si colocamos la ordenDisallow: /feed/$
. WARNING...! -
Well, I think that the above is well done, duplicated content can indeed hurt any website. Good, high-quality content is what Google is looking for, not otherway around. If you are unsure if what askapache.com is trying to establish, simply take a closer look at their #1 PageRank, #2 SERP and you'll get an immediate answer if this is good or not. Thanks, Emil SEO Agent
-
Your Robots.txt will block practically the whole site. Robots.txt does not take variables into consideration, so when you do a:
Disallow: /*?* Disallow: /*?
Its the same as :Disallow: /
Which blocks everything. I suggest you visit http://www.robotstxt.org/faq/robotstxt.html it will explain how wildcards are not supported. -
May I please ask you why you put this line : Disallow: */comments I guess it is to prevent specific comment URL from being indexed but the format of the comments URL isn't like this, isn't it?
-
I was wondering why the robots.txt file in this example is different to the one at askapache.com/robots.txt nice blog btw.
-
Hi Matt, I am not handy in Robots.txt file, but would i block whole pages from search engines incase if i use both your sample robots.txt file and php codes that you provided for WordPress.
-
Thanks for the post, it's really useful! But could you please tell more about following strings:
Disallow: (without any symbol after colon) Allow: /* Disallow: /
What do they mean? Is there any difference between "Allow: /*" and "Disallow:"? And how I should disallow indexing of a particular directory: "Disallow: /wp-admin" or "Disallow: /wp-admin/"? (should I use slash at the end or not?) -
Hay que limitar aceso a la carpetas para lascuales nos interesa limitar el rastreo, sin embargo cuidado en no occurir en el Blackhat, algunos manipulan los CSS, y limitan el aceso al buscador para que no se de cuenta de la adaptacion de los H1...Hx... yo propongo uno como : http://www.vuelomania.com/robots.txt
-
I fixed it. I put it in my root directory!
-
Why you are not disallowing /2007, /author and /page ???
-
Hello, my robots.txt is the following
Sitemap: http://www.xxx.es/sitemap.xml User-agent: * Disallow: /wp- Disallow: /search Disallow: /feed Disallow: /comments/feed Disallow: /feed/$ Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Disallow: /*/*/feed/$ Disallow: /*/*/feed/rss/$ Disallow: /*/*/trackback/$ Disallow: /*/*/*/feed/$ Disallow: /*/*/*/feed/rss/$ Disallow: /*/*/*/trackback/$ Disallow: /?s= Disallow: /dogs Disallow: /archives Disallow: /page Disallow: /author Disallow: /2007 Disallow: /category Disallow: /2008 Disallow: /2009 Disallow: /?livehit=
My issue is that my post only ranks when are on the homepage, is something wrong with my robots.txt ??? I´m ussing the same robots.txt on other two blogs and ranks really well -
Nice writeup. Thnx
-
Thanks for the great sharing, I am studying it now and plan to implement to all my blog sites.
-
great list, thanks!
-
If I have 10 pages website, Do we need to add Robots index,follow on each page? Or I need to add this only on index or default page so that robots can follow all links from there? Also what will happen if robot lands on a inner page first? does this line helps re-directing robot to follow links from index page?
-
is it good to have the tag cloud crawled. since it is a way of humanly categorizing content? I hear this is good, but should I then block my original wordpress categories?
-
I usually use these codes: User-agent: * Disallow: /cgi-bin Disallow: /wp-* I think I need some change. Thanks 4 your post:)
Add Comment!
My Picks
- Mod_Rewrite Tips and Tricks
- Crazy Advanced Mod_Rewrite
- THE Ultimate Htaccess
- THE Mod_Rewrite Cheatsheet
- Log Viewing in Console
- Custom php.ini Tips
- DNS: Round Robin Speed
- Fsockopen: PHP Socket-Level
- Rsync and SSH Transfer
Related Articles
- Allow AdSense Crawler Access
- WordPress robots.txt SEO
- Redirect index.php to root
- Robots.txt Secrets From Matt Cutts
- SEO Secrets of AskApache Part 2
- Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking
- Boost SEO, Drive Traffic with the 404 Error Page Plugin
- Updated: WordPress RewriteRules Viewer Plugin
- Crazy Cache WordPress Plugin Released
Newest Posts
- PirateBay and Anonymous SOPA Press Releases
- K-Swiss MFCEO Internet Commercial
- Drug Dealer Girl
- Htaccess Rewrites for Moving Urls
- Programming Fonts for the Web
