Instruct Search Engines to come back to site after you finish working on it

Instruct Search Engines to come back to site after you finish working on it

« Faster POST and GET Form SubmissionsRemoving Category Base from WordPress URLs »

What do you think Googlebot and other Search Engines do when they try to reach your site while you are tinkering with it?

Hopefully you aren’t doing anything that could slow the response time for the page google is trying to reach, and if google gets a 404 Not Found error or a 500 Error than your pagerank for that page could cease to exist!

What if you could conveniently tell Googlebot and other bots that you are working on the page but you would like them to come back in, oh, say an hour? I know what I did when I found out this was possible.. I found out how to do it and now I’m sharing with you.

Google Webmaster Central Blog

All About Googlebot

If my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the “down for maintenance” page?

You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.

How to use this SEO knowledge

We will use a bit of mod_rewrite code in a .htaccess file to send google and other search engine bots the 503 Service Temporarily Unavailable header, and we will also send a Retry-After: 3600 header to instruct the bots when they should re-check our page in 3600 seconds, (1 hour) to see if the page available

Retry-After Header

Article: Retry-After, 503 Service Unavailable

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of using Retry-After:

Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
Retry-After: 120

In the latter example, the delay is 2 minutes.

Send 503 only to Google Bots

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
# or RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule .* /cgi-bin/error/503.php

Send everyone except the developer a 503

Where REMOTE_HOST below is the developer’s IP address.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
 
RewriteCond %{REMOTE_HOST} !^1\.1\.1\.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503\.php [NC]
RewriteRule .* /cgi-bin/error/503.php

Send BOTS a 503; humans to error page.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503\.php [NC]
RewriteRule .* /cgi-bin/error/503.php
 
RewriteCond %{REMOTE_HOST} !^1\.1\.1\.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/404\.php [NC]
RewriteRule .* /under-development-explain.html [R=302,L]

What’s a 503 Service Temporarily Unavailable Header?

503 Service Temporarily Unavailable

The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

503 script code

503 Header with PHP

If you are using a CGI version of PHP than the Status header below is required in addition to the HTTP/1.1 header, otherwise if you are using mod_php than you do not need the Status header.

<?php
ob_start();
header('HTTP/1.1 503 Service Temporarily Unavailable');
header('Status: 503 Service Temporarily Unavailable');
header('Retry-After: 3600');
header('X-Powered-By:');
?><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Temporarily Unavailable</title>
</head><body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

503 Header with Perl CGI

#!/usr/local/bin/perl
 
print "Status: 503 Service Temporarily Unavailable\n";
print "Content-Type: text/html; charset=UTF-8;\n";
print "Retry-After: 3600\r\n\r\n";
 
print "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>503 Service Temporarily Unavailable</title>\n";
print "</head><body>\n<h1>Service Temporarily Unavailable</h1>\n<p>The server is temporarily unable to service your\n";
print "request due to maintenance downtime or capacity\nproblems. Please try again later.</p>\n</body></html>";

What are URL unreachable errors?

Google encountered an error when trying to access this URL. We may have encountered a DNS error or timeout, for instance. Your server may have been down or busy when we tried to access the page. Possible URL unreachable errors include:

1. 5xx error
2. 503 Network Unavailable
3. DNS issue
4. robots.txt file unreachable
5. Network unreachable

« Faster POST and GET Form Submissions
Removing Category Base from WordPress URLs »

MP3 Caper