FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  SEO  »  Instruct Search Engines to come back to site after you finish working on it

by 12 comments

What do you think Googlebot and other Search Engines do when they try to reach your site while you are tinkering with it?

Hopefully you aren't doing anything that could slow the response time for the page google is trying to reach, and if google gets a 404 Not Found error or a 500 Error than your pagerank for that page could cease to exist!

What if you could conveniently tell Googlebot and other bots that you are working on the page but you would like them to come back in, oh, say an hour? I know what I did when I found out this was possible.. I found out how to do it and now I'm sharing with you.

Google Webmaster Central Blog

All About Googlebot

If my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the "down for maintenance" page?

You should configure your server to return a status of 503 (network unavailable) rather than 200 (successful). That lets Googlebot know to try the pages again later.

How to use this SEO knowledge

We will use a bit of mod_rewrite code in a .htaccess file to send google and other search engine bots the 503 Service Temporarily Unavailable header, and we will also send a Retry-After: 3600 header to instruct the bots when they should re-check our page in 3600 seconds, (1 hour) to see if the page available

Retry-After Header

Article: Retry-After, 503 Service Unavailable

The Retry-After response-header field can be used with a 503 (Service Unavailable) response to indicate how long the service is expected to be unavailable to the requesting client. This field MAY also be used with any 3xx (Redirection) response to indicate the minimum time the user-agent is asked wait before issuing the redirected request. The value of this field can be either an HTTP-date or an integer number of seconds (in decimal) after the time of the response.

Retry-After  = "Retry-After" ":" ( HTTP-date | delta-seconds )

Two examples of using Retry-After:

Retry-After: Fri, 31 Dec 1999 23:59:59 GMT
Retry-After: 120

In the latter example, the delay is 2 minutes.

Send 503 only to Google Bots

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
# or RewriteCond %{HTTP_USER_AGENT} ^.*google.* [NC]
RewriteRule .* /cgi-bin/error/503.php

Send everyone except the developer a 503

Where REMOTE_HOST below is the developer's IP address.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

RewriteCond %{REMOTE_HOST} !^1.1.1.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503.php [NC]
RewriteRule .* /cgi-bin/error/503.php

Send BOTS a 503; humans to error page.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/503.php [NC]
RewriteRule .* /cgi-bin/error/503.php

RewriteCond %{REMOTE_HOST} !^1.1.1.1
RewriteCond %{REQUEST_URI} !^/cgi-bin/error/404.php [NC]
RewriteRule .* /under-development-explain.html [R=302,L]

What's a 503 Service Temporarily Unavailable Header?

503 Service Temporarily UnavailableThe server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.

503 script code

503 Header with PHP

If you are using a CGI version of PHP than the Status header below is required in addition to the HTTP/1.1 header, otherwise if you are using mod_php than you do not need the Status header.



503 Service Temporarily Unavailable



Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

503 Header with Perl CGI

#!/usr/local/bin/perl

print "Status: 503 Service Temporarily Unavailablen";
print "Content-Type: text/html; charset=UTF-8;n";
print "Retry-After: 3600rnrn";

print "nn503 Service Temporarily Unavailable

n";
print "n

Service Temporarily Unavailable

n

The server is temporarily unable to service yourn"; print "request due to maintenance downtime or capacitynproblems. Please try again later.

n";

What are URL unreachable errors?

Google encountered an error when trying to access this URL. We may have encountered a DNS error or timeout, for instance. Your server may have been down or busy when we tried to access the page. Possible URL unreachable errors include:

  1. 5xx error
  2. 503 Network Unavailable
  3. DNS issue
  4. robots.txt file unreachable
  5. Network unreachable

Tags

February 28th, 2008

Comments Welcome


Related Articles


My Online Tools
Popular Articles


Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman









[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain