This is part 2 of the exhaustive Htaccess Tutorial. I realized it was so lengthy that search engines and visitors were having real problems with it, so I moved half of it here. But this gave me the opportunity to add a ton of new stuff that I hadn't been able to add to the main htaccess tutorial. And now this new part 2 is already twice as big as the original! To sum up, this is a work in progress.

Moving Half of Original Tutorial to a new URL - SEO Rewriting

Well, the original tutorial was at /htaccess/apache-htaccess.html, but it was such a huge article that search engines were dropping it! So I split it into 3 new urls.

1. /htaccess/htaccess.html - 1
2. /htaccess/htaccess-rewrites.html - Rewrites
3. /htaccess/apache-htaccess-2.html - Part 2

The /htaccess/apache-htaccess-2.html is a lot of unfinished new stuff, and the /htaccess/htaccess-rewrites.html (this very page) still needs to be split several times to get the filesize down.

301 Link Juice

The /apache-htaccess.html url has been around since 2006, and it was full of original ideas and examples for using htaccess. Almost every htaccess guide or tutorial published since then has many of the examples and ideas from that tutorial, (I encourage people to modify and republish everything on this site according to copyright). I know it, the authors sometimes know it, but the main point is Google for sure knows it. That's the beauty of creating unique content, Google sees that. So that /apache-htaccess.html link has extreme link juice, from all the sites, books, papers, and presentations that have linked to it since 2006.

Filesize Is important

I like to use my blog as a way to keep notes about my research organized, and I am extremely good at doing research, unfortunately, that means I have a huge article. The filesize for the html alone is larger than all the other resources like javascript and images, combined.

• So that means it is very difficult to view on a mobile device, or a slow connection.
• For google-bot and other search engine crawlers and robots, this is a huge problem (I made a big mistake letting it get that size).
• The robot has to parse an enormous single html file, containing hundreds/thousands of external and internal links, and its such an issue they can decide to just skip indexing that url until the filesize is manageable.
• Once again, it's all about the human experience, a huge single file is not good for anyone who isn't printing it out to read offline.

Starting Fresh without losing juice

So since the page wasn't being indexed much since it was so huge, I decided to split up the content into new separate urls and utilize a 301 Redirect to transfer all the link juice from /apache-htaccess.html to /htaccess.html.

301 Redirect Timeline

1. Now, the idea is to take the first pages from the original multi-page guide, do a little improvement on that content, and save it to the new url /htaccess.html.
2. Then just continue taking the next page from the remaining original guide and creating new pages using the original links structure. /apache-htaccess-2.html, /apache-htaccess-3.html, etc.. This is a secondary backup to the new /htaccess.html url, which will receive the 301 link juice from the old url, but these secondary pages will help keep the links on external sites good.
3. Then, I setup a 301 Redirect in my .htaccess file to redirect the old url to the new /htaccess.html url.
4. Finally, I delete the old url and it is replaced forever by a 301 Redirect pointing to my new location, filesize problems eliminated.

301 Redirect Code Used

This is so easy to do with RedirectMatch, way faster and easier than using mod_rewrite to handle this, and much less overhead. Note that this is a general command that I will leave up for a few weeks and slowly tighten it up by looking at my Google Analytics and Apache Logs. For instance, this first redirectmatch rule is an older RedirectMatch I still have active to redirect all the old links pointing at /2006/htaccess/apache-htaccess.html to /htaccess/apache-htaccess.html from when I ditched the date-based permalinks back in 2007, and you can see it is a little tighter than the 2nd one which also redirects requests for apache-htaccess.html/feed/ or trackback or whatever.

RedirectMatch 301 ^/2006/.*apache-htaccess.html$ http://www.askapache.com/htaccess/apache-htaccess.html
RedirectMatch 301 ^/.*apache-htaccess.html(.*)$ http://www.askapache.com/htaccess/htaccess.html$1

Advanced 301 Redirects for SEO

I wrote a couple of articles that go into detail about maximizing the SEO with linking and redirects, it remains one of my most helpful articles to anyone trying to rank higher the right way, the Google way.. SEO Secrets of AskApache Part 2

Htaccess Rewrites for Moving Urls originally appeared on AskApache.com

Advanced Web Development by AskApache is a Firefox Collection I created for myself to make finding and installing my favorite Firefox Add-ons simple and easy.

My Setup

As this screenshot shows, I only use a handful of add-ons at a time. These buttons allow me to clear the DNS/Cookies/Cache for whichever site I'm on when I click it. Very very helpful for me as a web designer. The 4th button there is just a restart button. Other than those, Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.

I like the idea of the last.fm but it's not as powerful as the site, which is awesome. Lately I've been listening to Kings of Leon Radio...

Slow Firefox

The more add-ons you have, disabled or not, the slower Firefox is. (unless you are running your profile folder in TMPFS). Also, bookmarks and settings like that have a performance hit. I have been using Firefox since it launched way back when, and I have always kept my bookmarks when moving to new machines and new installations.. So with over 5 thousand bookmarks I finally did some debugging and discovered that was a huge cause of Firefox acting slow. Now I am trying to migrate them all over to Google's Gmarks, which knowing Google will be awesome eventually.

Multiple Firefox Installations, Sorta

The solution to all these problems is to use separate Firefox profiles which use separate folders to store your profile-specific extensions and settings. So I have profiles with upwards of 40 Addons installed and enabled, and another profile that is built for speed... It's very very slow to be running Firebug and have multiple tabs open.. You can use the profilemanager to load them specific profiles from the command line. I personally use separate icons on my Windows Quick Launch that I just modified the shortcut pointing to firefox to also have the profile commandline. Note also that you can have multiple profiles open and running simultaneously.. This lets you do some crazy networking and other random hacks like having many Firefox instances running each of which is configured to use a separate SOCKS Proxy or network interface, so you can really open up those pipes for some intense txrx.

You can do some very powerful things with Firefox that most people are unaware of, if you are interested start with these:

• Setting up an extension development environment
• Firefox Command Line Options

About This Collection

Web Development Add-ons for Advanced Web Developers. I personally use these to work on Sites, Servers, WordPress, Javascript, PHP, CSS, XHTML, validation, page-loading, SEO, optimizing, and much more. Add this collection.

Created by:AskApache

Updated:September 22, 2009

Share this Collection

• Digg this!
• Post to Facebook
• Add to Delicious
• Post to MySpace
• Share on FriendFeed
• Post to Twitter

Firefox Add-ons for Web Developers originally appeared on AskApache.com

In part I: SEO Secrets of AskApache.com we talked about content and building a website to be your "SEO Base". This article discusses some advanced SEO concepts to get a site indexed, move your best pages higher in search results, and controlling the pagerank/seo-juice of your site. but relatively easy ways to control and tweak WHAT urls on your site are indexed, and HOW. I've heard some people refer to this as "controlling pagerank flow" and "controlling pagerank juice", basically we want our best pages to rank higher in the search engine results.

Big Picture: Going from no website to AskApache.com in less than a year can be accomplished by anyone with unique content and a resolve to avoid any shortcuts and take it one step at a time.

We want what Google wants, to provide the most relevant content for someone who is doing a search. Basically, you want every url on your site that has unique content to be included in the index. In other words, you have to think like a search engine!

The Goal of Google

Here's what I mean, Google approaches search with the overwhelming goal of bringing the content to a searcher that is the most-likely to be what that searcher is searching for. Another way of looking at it is something I read on Google...

Google's goal is to get you off of their site as fast as possible by providing you with exactly what you are looking for.

Think about SEO like this

If you search for htaccess tutorial for seo on Google, would you be more likely to visit a tutorial about using .htaccess for seo or a category page for htaccess articles? AskApache.com has both of those urls included in the index, but the article ranks higher than the category page, as it very well should.

Get Your URLS Indexed

Before I explain how I am able to help Google and other search engines rank my article pages higher than my category pages, we need to get the urls in the index or nothing will show up. There are many well-discussed methods for getting included in the index, so I'll just list a few that I use.

1. Provide High-Quality original content, people will link to it
2. Get a sitemap and use Google Webmaster Tools
3. Publish an RSS/Atom Feed and ping the net when you publish a post
4. Use robots.txt and robots meta tags

Your URLs in the index

Here's how to find out which of your pages are indexed.

• Indexed pages in your site: site:
• Pages that link to your site's front page: link:
• The current cache of your site: cache:
• Information about your site: info:
• Pages that are similar to your site: related:

Break It Down

Yo homeslice! I didn't mean break dance.. I mean lets simplify AskApache in the context of getting our urls indexed high/low. Here's the stats: 1 Homepage, 206 Articles, 19 Pages, 31 Categories

1 Homepage

This page is generally the highest ranking page in the index, it should contain links to your best urls, and provide easy navigation

206 Specific Topic Articles

These are the article's (like this one) of AskApache.com and are the main source of search engine traffic. You want each url (if its a good article) to be ranked as high as possible. Some keys are to really make each article specific to a topic by using best-practice (X)HTML.

19 Static Pages

Most of these are pages like the online-tools hosted on this site, or other basic pages like about, contact us, site-map, etc.. Some of these you may want to rank very high ( like the /about/ page ) and some you may not want to even be included in the index.

31 Specific Topic Category Pages

These are tricky because they are generally just lists of articles from each category, which isn't specific enough to get much seach-engine-traffic, but is very useful to site visitors. I beefed up my category pages by adding additional information about the category topic in addition to excerpts of the articles.

Higher Pagerank = Higher Up in Search Results

So Googlebot and other search engine robots have these crazy complicated algorithms (many patented) that SEO Industry types may get caught up in and try to technically analyze them. I'm sure you've seen/read/heard the complicated advice that will always be pushed by many... advice like:

• analyze the number of words in the description/title/1st paragraph/etc.
• Make sure your "keyword" is sprinkled throughout the text every 10-30th word..
• Other equally unexciting technical analysis

Now if you've had success with that then props to you, success is success, but I personally choose to completely ignore all that. The number 1 thing that the top search engines advise is to design your page for a Human Visitor, not a computer. The golden rule for me is how I would rank the page, not how some algorithm would.

Designing for a Human Visitor

This is a major factor in your site being at the top vs. nowhere. You design your HTML to be as minimal as possible (see source code for my homepage) and contain ONLY the neccessary elements. Above all, use semantically sound XHTML markup. (view source of W3C)

External CSS/Javascript

Get your javascript and CSS out of your HTML and use external files (like this site) ALWAYS! You should start with just the HTML, no css, no colors, no javascript, and THEN you add the .css and then you add the javascript.

Site Accessibility

Say your browser didn't have a mouse, didn't support images, css, javascript, or even colors! Your HTML should be structured such that your page is still easily readable and easy to navigate. You can use lynx, links, and many other terminal-based browsers to test for this... please see the Web Accessibility Initiative (WAI) for detailed info.

Web accessibility refers to the practice of making websites usable by people of all abilities and disabilities. When sites are correctly designed, developed and edited, all users can have equal access to information and functionality. For example, when a site is coded with semantically meaningful HTML, with textual equivalents provided for images and with links named meaningfully, this helps blind users using text-to-speech software and/or text-to-Braille hardware.

Controlling a URL's Pagerank

A few tools and techniques are available for controlling the "juice" or "pagerank" of your urls.

1. Robots.txt
2. Robots Meta Tags
3. Links

Robots.txt

I've done quite a bit of research and experimentation with robots.txt files, which is a file located in the root of your website at http://www.askapache.com/robots.txt that is downloaded by all legitimate search engine spiders/bots and used as a Blacklist to prevent certain urls from being indexed. Here are a few of the articles on this site, which you may skip if you like as they don't illustrate the big-picture that I am going to discuss now.

• Control Flow of Pagerank with robots.txt and NoFollow, NoIndex
• WordPress robots.txt
• SEO with Robots.txt
• Google AdSense using robots.txt
• WordPress robots.txt file optimized for SEO and Google

How To Use Robots.txt

Even though robots.txt files are for whitelisting and blacklisting urls, I have found that they should only be used as an extreme form of blacklisting. When you Disallow a url in your robots.txt file, that means most search engine bots won't even LOOK at the url. As you can see in the below example, I only disallow urls that shouldn't ever be LOOKED at. The real powertool is the robots meta tag.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
 
Sitemap: http://www.askapache.com/sitemap.xml

Robots Meta Tag

Ok I'm really trying to simplify, because what you should understand is the big-picture. Every page can have a robots meta tag in the header, and this robots meta tag can tell the search-engine to index/not-index AND follow/not-follow. Here are some examples:

<meta name="robots" content="index" />
<meta name="robots" content="noindex" />
<meta name="robots" content="noindex,follow" />
<meta name="robots" content="index,nofollow" />

content="index" / noindex

index means the search engine is free to index, archive, cache, and follow the page whereas noindex means DO NOT include this page in the search engine results.

content="follow" / nofollow

follow means the search engine is free to LOOK at the page and follow the links on the page whereas nofollow means DO NOT follow the links on the page.

WordPress Auto-Robots meta tag code

Just add this to any plugin file and it will add the right robots meta tag to your site.. tweak to taste.

function askapache_robots_header(){
 global $wpdb;
 
 $robot = '<meta name="robots" content="noindex,follow,nocache,noarchive" />';
 
 if ( is_paged() || is_search() || is_404() || is_author() || is_tag() )
   $robot = '<meta name="robots" content="noindex,follow" />';
 elseif ( is_home() || is_front_page() || is_single() )
   $robot = '<meta name="robots" content="follow,index" />';
 elseif ( is_category() || is_page() )
   $robot = '<meta name="robots" content="follow" />';
 
 echo $robot . "\n";
}
add_action( 'wp_head', 'askapache_robots_header' );

Links

External and Internal Links are the crux of SEO. It's important to start FIRST on your Internal Links and linking structure... Once you are satisfied that the correct pages are indexed and ranked appropriately, then you can begin to look at external links.

• The fewer links on a page, generally the better.
• If every page of your site points to the same url on your site, pagerank goes up from the number of internal links.
• Use of the rel, title, alt attribute semantically is very helpful. (next, prev, index)
• Put your best links higher up in the XHTML, and put helpful/solid links at the end.
• You can add rel="nofollow" to links that you dont want followed.

The web has gotten to be so full of malicious/non-helpful SEO activity that I recommend developing your content NOT external links. If you want to do this right and provide great content that makes search engine users happy and makes the web better, then explore this blog and develop content until the next article in this series, where I'll show you how to make your site explode.

Stay tuned for Part III, which will dive deeper into the pipeworks of AskApache.com

SEO Secrets of AskApache Part 2 originally appeared on AskApache.com

First let me say that I don't read SEO stuff, I don't participate in the SEO community, I only have an interest in regards to the technology used by the search engines. All websites are hosted on servers, mostly Apache, and that is the primary topic of this blog. During the past year of this blog, my research into non-seo areas has turned up some very valuable SEO techniques.. All of them legal, ethical, and genuinely good for the Internet at large.

Update: SEO Secrets part II: Advanced Indexing and Pagerank Control

Some Background

I started this blog in late 2006, my first foray into blogging, and I've been extremely successful at achieving top ten google rankings and maintaining on average 15K unique visitors/day (per google analytics) 85% of which come from search engine traffic.

NOTE: I take it for granted that anyone reading AskApache is an expert of some skill, if you aren't I apologize, I can't waste time on the easy stuff.

Prerequisite SEO

There are literally hundreds of thousands of SEO articles on the net, 99.9% of which are absolute garbage. Especially in the sense that they just repeat the same 10 year old stuff. However, to do any kind of advanced SEO like I am going to discuss in this article, I am assuming that you, the intelligent reader, has already read those and has a basic understanding of SEO fundamentals like meta tags, titles, keywords, etc.

First, Great Content

The foremost and most important step in achieving any kind of traffic is to produce great content. I'm sure you've heard that a million times, but let me break it down how I perceive it. Before I even started to mess with SEO for AskApache.com I began by writing articles. At that point I didn't have a clue what my blog was going to be about or even if I was going to be doing it after a week.

What is Content

For me, being a top-paid professional web developer, I spend about 80% of my time doing research. I think that is a bit uncommon, but its a throwback from the 10 years I spent in the network/computer security field, where research is 99% of the job, a story for another time perhaps.

So the research I was doing at that time was about best-practice standards-based web design, mainly XHTML Strict, CSS, and unobtrusive javascript. Each of those subjects has become near and dear to my heart, and each should also be mandatory learning for anyone interested in SEO. The best advice I can give towards that end is checking out the CSS, Javascript, and XHTML Strict source code for this page and site. And of course the holy W3.org.

In addition to striving to master those 3 subjects, I was also and always will be researching web programming languages like PHP, Ajax, Ruby, and Server Technology like Apache. Although I should note that my research into Apache and server technologies is more of a hobby than a job requirement, also a throwback to my days in the security industry and of course my love for open source software.

My Content

So basically I was spending 25% of my time at work actually working, and the other 75% of the time I would research how to do something better, faster, the best. Incredibly, I discovered or re-discovered a ton of tips, tricks, and methods to aid me in my work. I was learning so much valuable information that I joined a couple of forums to discuss them and get feedback on making them even better. Soon I realized that I was one of a small few who actually post content to a forum instead of just questions, so I decided to write my tutorials down on a blog, and AskApache was born.

So that is why this blog is comprised of almost 100% tutorials, and why almost all of them are completely original works you won't find elsewhere. That's how I create content, but you might do something different. Whatever it is that you do for content, just make sure you are providing VALUE with everything you do. Not to everyone, just stuff that you would consider to have value if you were reading it.

Second, Great Site

Ok so I had 10 or so great articles that I knew would provide value for many web developers, but so what? Nobody cares you know.. That's when I decided to take a closer look at the software that was running my new blog, WordPress, and I've been hacking the code ever since on my never-ending quest to be the best and know the most advanced web development. You'll see why in a couple paragraphs.

I Mean, a Really Great Site

By great, I mean you need to make it incredibly user-friendly. Every design and development decision you make should be about the visitor. THATS the number one key to success on the net, regardless of endeavor. Here is a list of things you definately need to have before you do SEO, I'm not listing obvious stuff like descriptions, titles, and good writing.

1. Intuitive and circular, your website should be a spider-web of urls.
2. Easy to read, plenty of white-space, design is your decision but I like minimalistic.
3. Super fast rendering. You need effective caching and optimization.
4. A very helpful 404 error page, hopefully never seen.

Focus in on your URL's

Many sites that use a CMS of some kind, be it Drupal or WordPress, have hundreds or thousands of URL's even if they only have 10 actual posts/articles.

Removing Duplicate Content

You've all heard this before, but almost no-one has taken it to the level I am going to discuss. Bear with me.

Removing duplicate content is actually a very straightforward process if you know what you are doing, and if you don't, well that's why I'm going to quickly explain how to really do a good job.

Locate Duplicate Content and URLS

People misunderstand that you should just not repeat the same paragraph in a different article, that is partially true, but the main impact this has on your site is if you can access the same article from more than a single URL.

I hope you realize you MUST use pretty urls like my site and not codey looking ones with question marks. You can find any potential duplicate urls on wordpress with the rewriterules plugin. Also look at Google's webmaster tools to look for any duplicate urls, and you can use xenus link sleuth tool as well.

Remove Duplicate Urls with .htaccess

Once you've found duplicate urls, you need to instruct google and other search engine robots to be redirected to the correct url. By doing a 301 redirect you tell the search engines NOT to index the bad url, only the good one. Below are some of the .htaccess code I use on this site to accomplish this technique, this is gold I myself use so pay attention. It works.

301 Redirects with mod_rewrite

First lets start with one everyone should know, and the most common, to www or not to www?

RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Its a highly rare individual who has seen this one, which forces requests for .html/ to .html

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /[^\.]+\.html/\ HTTP/ [NC]
RewriteRule ^(.*)\.html/$ http://www.askapache.com/$1.html [R=301,L]

301 Redirects without mod_rewrite

For the rest of the duplicate urls that you find, I like to use Redirect and RedirectMatch.

This redirects requests that start with an & or /&amp or /( to my homepage.

RedirectMatch 301 ^/&(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/&amp(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/([\(]+)(.*)$ http://www.askapache.com/

This redirects requests with //whatev to /whatev

RedirectMatch 301 ^//(.*)$ http://www.askapache.com/$1

But this is just a brief look at what you will have to spend some time on. There are detailed guides to doing this with mod_rewrite and using Redirect on my blog. Its time now for some real SEO tips. The heart of the matter, as it were.

Wrap It Up

So I realize that was brief, so I want to really stress 2 things or you won't take away much from part 1.

Locate Duplicate URLS

This is truly one of the most important things in my personal experience. I personally take this as extreme as I can, I regularly grep my access files, mod_security and error log files looking for bad URLS. I am always checking them out to see if someone has a bad link to me somewhere, or if someone just typed it in wrong. If its a bad link on a site, I will very politely attempt contacting the webmaster about it until they fix it.

Even I, with my many colorful years of Internet travel, was caught off-guard by the variety and creativity and the sheer number of urls people are using to link to my site. I found that often bad links would be published because my URL was just too long, so I shortened the URL's. Now of course bad links can't really even touch my site with all my 301's in place.

Besides grepping your server's logs, the 2nd best place to locate duplicate urls or just plain wrong urls is by using Googles free webmaster tools. They keep track of all the bad urls linking to your site and allow you to download this data in a .csv spreadsheet format. The first time I checked into this I found over 1,000 bad links, after a couple months with my RewriteRules and 301 Redirects, I've narrowed the list down to under 50 most months. That is a powerful reason to use 301 Redirects, as we'll really get into in part 2.

301 Redirect Bad URLS

Finding the bad urls takes some time, a couple hours even, and then the whole reason you do that is to be able to create 301 Redirects for all of those bad urls to good urls.

One reason that I wasn't even aware of until several months ago is that when Googlebot locates a bad URL for your site, it tries to access it, and if you haven't planned for this in advance, your page most likely will return a 200 OK status header, or if you are lucky a 404 Not Found error, both of which really hurt you.

Basically, a 200 response will produce duplicate content in 99.9% of the time, and 404 responses will whisper to Google's algorithms that you don't know what you are doing and your site isn't authoritative. 200 means google will index your site, 404 means google won't index your site, but it also won't give up trying for awhile, which takes away from your real urls.

What a 301 Response tells Google

301 Responses were practically invented for user-agents/clients/web-crawling robots like google. They instruct the client, whether that be a persons browser or a googlebot, that the resource/page that they are looking for is actually at a different URL. This is an authoritative response that makes googlebot and other search engines ecstatic because now they can give up on the 200 and 404 responses that didn't really give them an answer either way.

On the other hand, a great 404 can and should be just as powerful as a 301, but hardly anyone uses them in the correct way according to HTTP 1.1 or 1.0 Specifications. We'll tear that subject apart further down the road.

I'll leave this topic for now with one last idea, 301 Redirects when implemented and used correctly, actually redirect the page rank and search engine ranking for itself to the redirected / correct url. That means if you have 1000 unique links pointing to your article, and all of them are incorrect in some way, if you can 301 redirect all of those bad links to your correct link you now have 1000 new good links! It has to be done right and in a classy way though of course.

Now that you have content and a great site, its time to SEO like a mofo.

SEO Secrets of AskApache.com originally appeared on AskApache.com

What do you think Googlebot and other Search Engines do when they try to reach your site while you are tinkering with it?

Hopefully you aren't doing anything that could slow the response time for the page google is trying to reach, and if google gets a 404 Not Found error or a 500 Error than your pagerank for that page could cease to exist!

What if you could conveniently tell Googlebot and other bots that you are working on the page but you would like them to come back in, oh, say an hour? I know what I did when I found out this was possible.. I found out how to do it and now I'm sharing with you.

Google Webmaster Central Blog

All About Googlebot

If my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the "down for maintenance" page?

You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.

How to use this SEO knowledge

We will use a bit of mod_rewrite code in a .htaccess file to send google and other search engine bots the 503 Service Temporarily Unavailable header, and we will also send a Retry-After: 3600 header to instruct the bots when they should re-check our page in 3600 seconds, (1 hour) to see if the page available

Retry-After Header

Article: Retry-After, 503 Service Unavailable

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of using Retry-After:

Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
Retry-After: 120

In the latter example, the delay is 2 minutes.

Send 503 only to Google Bots

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
# or RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule .* /cgi-bin/error/503.php

Send everyone except the developer a 503

Where REMOTE_HOST below is the developer's IP address.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
 
RewriteCond %{REMOTE_HOST} !^1\.1\.1\.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503\.php [NC]
RewriteRule .* /cgi-bin/error/503.php

Send BOTS a 503; humans to error page.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503\.php [NC]
RewriteRule .* /cgi-bin/error/503.php
 
RewriteCond %{REMOTE_HOST} !^1\.1\.1\.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/404\.php [NC]
RewriteRule .* /under-development-explain.html [R=302,L]

What's a 503 Service Temporarily Unavailable Header?

503 Service Temporarily Unavailable The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

503 script code

503 Header with PHP

If you are using a CGI version of PHP than the Status header below is required in addition to the HTTP/1.1 header, otherwise if you are using mod_php than you do not need the Status header.

<?php
ob_start();
header('HTTP/1.1 503 Service Temporarily Unavailable');
header('Status: 503 Service Temporarily Unavailable');
header('Retry-After: 3600');
header('X-Powered-By:');
?><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

503 Header with Perl CGI

#!/usr/local/bin/perl
 
print "Status: 503 Service Temporarily Unavailable\n";
print "Content-Type: text/html; charset=UTF-8;\n";
print "Retry-After: 3600\r\n\r\n";
 
print "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Temporarily Unavailable</title>\n";
print "</head><body>\n<h1>Service Temporarily Unavailable</h1>\n<p>The server is temporarily unable to service your\n";
print "request due to maintenance downtime or capacity\nproblems. Please try again later.</p>\n</body></html>";

What are URL unreachable errors?

Google encountered an error when trying to access this URL. We may have encountered a DNS error or timeout, for instance. Your server may have been down or busy when we tried to access the page. Possible URL unreachable errors include:

1. 5xx error
2. 503 Network Unavailable
3. DNS issue
4. robots.txt file unreachable
5. Network unreachable

Instruct Search Engines to come back to site after you finish working on it originally appeared on AskApache.com

Finally some robots.txt questions have been been answered. The most interesting thing is how simple robots.txt files really are. And how incredibly useful they can be at directing your pagerank wherever you want it. The robots.txt secret is that you use the robots.txt file as the first and not too restrictive control. Then you use XHTML meta tags NoIndex and NoFollow in the <head> section of your html. Finally you mark up actual links in your source code by adding the rel="nofollow" to control pagerank flow.

These hounding robotstxt questings were answered by none other than Matt Cutts in this upbeat and not too technical interview done by Eric Enge of STC. The interview transcript is fairly long and touches on a multiple issues that are quite frankly out my league, so I just grabbed parts of the interview that were answering some robots.txt questions.

Eric Enge Does the NoFollow metatag imply a NoIndex on a page? Matt Cutts No. The NoIndex and NoFollow metatags are independent. The NoIndex metatag, for Google at least, means don't show this page in Google's index. The NoFollow metatag means don't follow the outgoing links on this entire page. ... Eric Enge Right. So there are two levels of NoFollow. There is the attribute on a link, and then there is the metatag, right. Matt Cutts Exactly. Eric Enge What we've been doing is working with clients and telling them to take pages like their about us page, and their contact us page, and link to them from the Homepage with a NoFollow attribute, and then link to them using NoFollow from every other page. It's just a way of lowering the amount of link juice they get. These types of pages are usually the highest PageRank pages on the site, and they are not doing anything for you in terms of search traffic. Matt Cutts Absolutely. So, we really conceive of NoFollow as a pretty general mechanism. The name, NoFollow, is meant to mirror the fact that it's also a metatag. As a metatag NoFollow means don't crawl any links from this entire page. NoFollow as an individual link attribute means don't follow this particular link, and so it really just extends that granularity down to the link level. We did an interview with Rand Fishkin over at SEOmoz where we talked about the fact that NoFollow was a perfectly acceptable tool to use in addition to robots.txt. NoIndex and NoFollow as a metatag can change how Googlebot crawls your site. It's important to realize that typically these things are more of a second order effect. What matters the most is to have a great site and to make sure that people know about it, but, once you have a certain amount of PageRank, these tools let you choose how to develop PageRank amongst your pages. Eric Enge Right. Another example scenario might be if you have a site and discover that you have a massive duplicate content problem. A lot of people discover that because something bad happened. They want to act very promptly, so they might NoIndex those pages, because that will get it out of the index removing the duplicate content. Then, after it's out of the index, you can either just leave in the NoIndex, or you can go back to robots.txt to prevent the pages from being crawled. Does that make sense in terms of thinking about it? ... Matt Cutts In general, Google does a relatively good job of following the 301s, and 302s, and even Meta Refreshes and JavaScript. Typically what we don't do would be to follow a chain of redirects that goes through a robots.txt that is itself forbidden. Eric Enge Let's talk a bit about the various uses of NoIndex, NoFollow, and Robots.txt. They all have their own little differences to them. Let's review these with respect to 3 things: (1) whether it stops the passing of link juice; (2) whether or not the page it still crawled; and: (3) whether or not it keeps the affected page out of the index. Matt Cutts I will start with robots.txt, because that's the fundamental method of putting up an electronic no trespassing sign that people have used since 1996. Robots.txt is interesting, because you can easily tell any search engine to not crawl a particular directory, or even a page, and many search engines support variants such as wildcards, so you can say don't crawl *.gif, and we won't crawl any GIFs for our image crawl. We even have additional standards such as Sitemap Support, so you can say here's a link to where my Sitemap is can be found. I believe the only robots.txt extension in common use that Google doesn't support is the crawl-delay. And, the reason that Google doesn't support crawl-delay is because way too many people accidentally mess it up. For example, they set crawl-delay to a hundred thousand, and, that means you get to crawl one page every other day or something like that. We have even seen people who set a crawl-delay such that we'd only be allowed to crawl one page per month. What we have done instead is provide throttling ability within Webmaster Central, but crawl-delay is the inverse; its saying crawl me once every "n" seconds. In fact what you really want is host-load, which lets you define how many Googlebots are allowed to crawl your site at once. So, a host-load of two would mean, 2 Googlebots are allowed to be crawling the site at once. Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results. Eric Enge Based on the links from other sites to those pages. Matt Cutts Exactly. So, we would return the un-crawled reference to eBay. Matt Cutts Exactly. The funny thing is that we could sometimes rely on the ODP description (Editor: also known as DMOZ). And so, even without crawling, we could return a reference that looked so good that people thought we crawled it, and so that caused a little bit of earlier confusion. So, robots.txt was one of the most long standing standards. Whereas for Google, NoIndex means we won't even show it in our search results. So, with robots.txt for good reasons we've shown the reference even if we can't crawl it, whereas if we crawl a page and find a Meta tag that says NoIndex, we won't even return that page. For better or for worse that's the decision that we've made. I believe Yahoo and Microsoft might handle NoIndex slightly differently which is little unfortunate, but everybody gets to choose how they want to handle different tags. Eric Enge Can a NoIndex page accumulate PageRank? Matt Cutts A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page. Eric Enge So, it can accumulate and pass PageRank. Matt Cutts Right, and it will still accumulate PageRank, but it won't be showing in our Index. So, I wouldn't make a NoIndex page that itself is a dead end. You can make a NoIndex page that has links to lots of other pages. Eric Enge Interviews Google's Matt Cutts For example you might want to have a master Sitemap page and for whatever reason NoIndex that, but then have links to all your sub Sitemaps. ... Eric Enge Another example is if you have pages on a site with content that from a user point of view you recognize that it's valuable to have the page, but you feel that is too duplicative of content on another page on the site That page might still get links, but you don't want it in the Index and you want the crawler to follow the paths into the rest of the site. Matt Cutts That's right. Another good example is, maybe you have a login page, and everybody ends up linking to that login page. That provides very little content value, so you could NoIndex that page, but then the outgoing links would still have PageRank. Now, if you want to you can also add a NoFollow metatag, and that will say don't show this page at all in Google's Index, and don't follow any outgoing links, and no PageRank flows from that page. We really think of these things as trying to provide as many opportunities as possible to sculpt where you want your PageRank to flow, or where you want Googlebot to spend more time and attention.

Matt Cutts Blog - Full Length Transcript at Stone Temple Consulting here, Or you can read Erics Blog Post about it.wordpress robots.txt

Robots.txt Secrets From Matt Cutts originally appeared on AskApache.com

Check out this free online SEO site-scoring tool. The SEO generated report is simple, neat, and helpful. The main thing I like about it is how it is a stand-alone website, a residual money making machine.

WebSite Grade For www.askapache.com : 98/100 (October 06, 2007)

A website grade of 98 for www.askapache.com means that of the thousands of websites that have previously been submitted to the tool, our algorithm has calculated that this site scores higher than 98% of them in terms of its marketing effectiveness. The algorithm uses a proprietary blend of over a dozen different variables, including search engine data, website structure, approximate traffic, site performance, and others.

Score Summary

Google PageRank : 5

"Google PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value. In essence, Google interprets a link from page A to page B as a vote, by page A, for page B. But, Google looks at more than the sheer volume of votes, or links a page receives; it also analyzes the page that casts the vote. Votes cast by pages that are themselves important weigh more heavily and help to make other pages important."
Learn More

Web Page Structure

To learn more about what meta-data is and why it is important, take a look at the following article:
SEO for Small Business Executives: Understanding the Magic Of Meta-Data

High Number Of Meta Keywords

There are a high number of keywords in the meta-data of the web page.
We believe that though the search engines don't weigh keywords as heavily as they used to, they're still important to get right. By using a high number of keywords, it is possible that you are diluting the effect of your most important keywords. We would suggest keeping the keywords to 10 or less. Currently, this site has 21 keywords in the meta-data.

Domain Info

Heading Summary

Similar to how newspapers and magazines use headings and sub-headings to help readers, websites can use special tags in their HTML. These tags not only help human readers read the content, they also help search engine spiders better understand the content on a page and what is most important. It is generally a good idea to use heading tags to help signal to the search engines, what the web page is about.

Image Summary

Images are a great way to enhance a website from a user's perspective. However, it is important to note that search engine crawlers cannot really "see" images. So, if you have lots of images that contain textual content within the image itself, this content will not be seen by the crawlers.

HTML helps address this issue by providing a way to specify the textual content for an image using the "alt" attribute. The alt attribute allows web pages to assign specific text as the "alternative" content for images for those that cannot view the images themselves. This can be search engine crawlers or text-only web browsers.

Google Indexed Pages : 671

Measures the approximate number of pages on http://www.askapache.com/ that have been stored in the Google index. The Google web crawler will visit the website periodically and look for new content and store it in its index. Generally, the more pages a website has within the Google cache, the better.

There are0pages from http://www.askapache.com/ in the supplemental results area of Google.
These pages will generally not rank on Google.com. A high number of supplemental pages may indicate a problem with the site.

Google Crawl Date : October 05, 2007 @ 06:12 PM

Google will periodically crawl websites looking for new and updated content. In general, it is desired to have the frequency with which Google crawls a site to be higher, rather than lower. Sites that get crawled more often have more frequent updates and more authority.

Conversion Methods

The WebSite grader attempts to measure both the ability for a website toattracttraffic and the ability to convert that traffic into qualified leads and prospects. The software searches for common conversion mechanisms such as "calls to action". Examples include RSS feeds, email subscription forms and other ways that site visitors can express interest in the offering.

Inbound Links

One of the most important measures for a website is how many other sites are linking to it. The more links the better.

Links Summary

Technorati Ranking : 18,487

Technorati is a popular website ranking service. It measures the popularity of a given website as compared to all other sites that have been submitted to its system.

This website currently has a Technorati ranking of 18,487.

del.icio.us bookmarks : 197

del.icio.us is a web-based bookmarking site. Delicious users save their bookmarks on the website so they can access them from any browser. A website that has many users bookmarking it is generally popular and will get more traffic. http://www.askapache.com/ has about 197 bookmarks stored.

Alexa Traffic Ranking : 32,619

Alexa is an online service that measures the approximate traffic for millions of sites on the Internet.

The website http://www.askapache.com/ has a ranking of 32,619. This means that it is the 32,619th most trafficed site on the web -- as measured by Alexa.

Readability Level : Advanced Degree (PhD)

This score measures the approximate level of education necessary to read and understand the web page content. In most cases, the content should be made to be simple so that a majority of the target audience can understand it.

Search Rankings

digg.com Submission Summary

The website www.askapache.com has had 100 articles submitted to digg.com. These articles received a total of 510 diggs (votes) and 0 of these articles made it to the front page of digg.com.

Website SEO Score Tool originally appeared on AskApache.com