This is part of the GRUB article on the Arch Linux wiki that I contributed. As of now, just a copy.

The grub menu.lst provides for a convenient way to add a number of entries with extended kernel parameters to configure all sorts of advanced settings to enable you to quickly and conveniently boot into your existing system with varying levels of debugging output. It's very easy and useful to create several levels of debugging just by adding additional entries to your grub configuration. And if you ever have issues or problems down the road due to a power-failure or hardware failure, it can save you hours of trouble, and of course nothing can beat debugging output when it comes to learning about your system.

Useful Menu.lst Entries

If you are interested in debugging, then you deserve some grub entries for powerusers, here are a few that I like (just add to your menu.lst).

title Shutdown the Computer
halt
 
title Reboot the Computer
reboot
 
title Command Line
commandline
 
title Install GRUB to hd0 MBR
root (hd0,0)
setup (hd0)
 
title Matrix
color green/black light-green/green
 
title Scan for /boot/grub/menu.lst
find --set-root --ignore-floppies /boot/grub/menu.lst
configfile /boot/grub/menu.lst
 
title Scan for /boot/menu.lst
find --set-root --ignore-floppies /menu.lst
configfile /boot/menu.lst
 
# http://www.vortex.prodigynet.co.uk/x86test/
title    Run x86test (CPU Info)
kernel /boot/x86test_zImage.bin
#wget http://www.vortex.prodigynet.co.uk/x86test/x86test_zImage.bin
 
# http://www.memtest.org/
title    Run memtest86+ (Memory Testing)
kernel /boot/memtest86+-1.70.bin

Light Debug

A quick way to see more verbose messages on your console is to bootup your normal grub entry after appending verbose to the kernel line. This simple word added to your kernel line turns on more logging thanks to the /etc/rc.sysinit file, which at the top of the file runs:

if /bin/grep -q " verbose" /proc/cmdline; then /bin/dmesg -n 8; fi

Very simple way to get a bit more messages and debug output in your logs.

title  Arch Linux DEBUG Light
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait verbose
initrd /kernel26.img

Medium Debug

This example menu.lst entry turns on real logging that is set by the kernel and not in an init script. Adding the debug kernel parameter to your kernel line is recognized by a lot of linux internals and enables quite a bit of debugging compared to the default.

title Arch Linux DEBUG Medium
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug
initrd /kernel26.img

Heavy Debug

An even more impressive kernel parameter is the ignore_loglevel, which causes the system to ignore any loglevel and keeps the internal loglevel at the maximum debugging level, basically rendering dmesg unable to lower the debug level.

title Arch Linux DEBUG Heavy
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug ignore_loglevel
initrd /kernel26.img

Extreme Debug

If the "Heavy Debug" seemed like a lot of output, thats about 1/2 of the logging that occurs with this example. This does a couple things, it uses the earlyprintk parameter to setup your kernel for "early" "printing" of messages to your "vga" screen. The ,keep just lets it stay on the screen longer. This will let you see logs that normally are hidden due to the boot-up process. This also changes the log buffer length to 10MB, and also instructs that any fatal signals be printed with print_fatal_signals. The last one, sched_debug, you can look up in the very excellent kernel documentation on kernel parameters.

title Arch Linux DEBUG Extreme
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug
initrd /kernel26.img

Insane Debug

The first few debugging examples showed some really nice kernel parameters to turn on really verbose debugging. This kind of debugging is absolutely critical if you want to max out your system or just learn more about what is going on behind the scenes. But there is a final trick that is my favorite, it's the ability to set both environment variables, and more importantly, module parameters at boot.

As an example, here is the boot line that I am using at the moment on an older Dell Desktop, just to illustrate module parameters and environment vars.

title  Arch Linux X-256
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait pause_on_oops=5 panic=60 i915.modeset=1 no_console_suspend ipv6.disable=1 TERM=xterm-256color quiet 5
initrd /kernel26.img

Since it's low on both memory and CPU, I disable ipv6. I also turn on kernel modesetting for the i915 video card, set my terminal to be xterm-256color, and boot straight into X. This lets me use a very optimized arch-linux configuration, amazing how fast thanks to using slim as the login manager, ratpoison as my window manager, and terminal with tmux as my login shell, all from boot, as the pstree shows (plus Synergy!).

init,1
  |-slim,3096
  |   |-X,3098 -nolisten tcp vt07 -auth /var/run/slim.auth
  |   `-ratpoison,3107,askapache
  |       |-terminal,5341 -x sh -c exec /usr/bin/tmux -2 -l -u -q attach -d -t tmux-askapache
  |       |   |-bash,11165
  |       |   |-tmux,5345 -2 -l -u -q attach -d -t tmux-askapache
  |       |   `-{terminal},5346
  |       `-xscreensaver,3113 -no-splash
  |-synergyc,6121,galileo -f --name galileo-fire --restart 10.66.66.2:26666
  |
  `-tmux,5348,askapache -2 -l -u -q attach -d -t tmux-askapache
      |-bash,5351
      |   `-ssh,9969 lug@askapache.com
      `-bash,5868
         `-vim,11149 -p sda1/grub/menu.lst /boot/grub/menu.lst

That kind of optimized system is only possible if you first can figure out your system, by debugging both the kernel as previously illustrated, debugging the init process, and most importantly, by debugging the modules enabled for your system's hardware/firmware/software. Debugging modules is challenging but worth the effort, and then you are able to do some truly insane debugging from grub like the following example, note that the actual grub entry is all on one line, but I split it into 4 lines so you could see it all. This basically turns on every module on this little Dell desktop to be at the absolute max debug level. There is so much logging when I boot this that the system grinds to a halt and is slower than a TI-89 calculator (See Improve Boot Performance).

title  Arch Linux DEBUG INSANE
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait ignore_loglevel debug debug_locks_verbose=1 sched_debug initcall_debug mminit_loglevel=4 udev.log_priority=8
       loglevel=8 earlyprintk=vga,keep log_buf_len=10M print_fatal_signals=1 apm.debug=Y i8042.debug=Y drm.debug=1 scsi_logging_level=1 usbserial.debug=Y
       option.debug=Y pl2303.debug=Y firewire_ohci.debug=1 hid.debug=1 pci_hotplug.debug=Y pci_hotplug.debug_acpi=Y shpchp.shpchp_debug=Y apic=debug
       show_lapic=all hpet=verbose lmb=debug pause_on_oops=5 panic=10 sysrq_always_enabled
initrd /kernel26.img

A couple key items from that grub entry are sysrq_always_enabled which forces on the sysrq magic, which really is a lifesaver when debugging at this level as your machine will freeze/stop-responding sometimes and it's nice to use sysrq to kill all tasks, change the loglevel, unmount all filesystems, or do a hard reboot. Another key parameter is the initcall_debug, which debugs the init process in excruciating detail. Very useful at times. The last parametery I find very useful is the udev.log_priority=8 to turn on udev logging.

Break Into Init

For instance, If you add break=y to your kernel cmdline, init will pause early in the boot process (after loading modules) and launch an interactive sh shell which can be used for troubleshooting purposes. (Normal boot continues after logout.) This is very similar to the shell that shows up if your computer gets turned off before it is able to shutdown properly. But using this parameter lets you enter into this mode differently at will.

title  Arch Linux Init Break
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait break=y
initrd /kernel26.img

Debugging init

This awesome parameter udev.log_priority=8 does the same thing as editing the file /etc/udev/udev.conf except it executes earlier, turning on debugging output for udev. If you want to know your hardware, that is the key parameter right there. Another trick is if you change the /etc/udev/udev.conf to be verbose, then you can make your initrd image include that file to turn on verbose udeb debugging by adding it to your {{Filename|/etc/mkinitcpio.conf} like:

FILES="/etc/modprobe.d/modprobe.conf /etc/udev/udev.conf"

, which on arch is as easy as

# mkinitcpio -p kernel26

Debugging udev is key because the initrd performs a root change at the end of its run to usually launch a program like /sbin/init as part of a chroot, and unless the new file system has a valid /dev directory, udev must be initialized before invoking chroot in order to provide /dev/console.

exec chroot . /sbin/init <dev/console >dev/console 2>&1

So basically, you aren't able to view the logs that are generated before /dev/console is initialized by udev or by a special initrd you compiled yourself. One method the kernel developers use to be able to still get the log messages generated before /dev/console is available is to provide an alternative console that you can enable or disable from grub.

Net Console

If you read through the kernel documentation regarding debugging, you will hear about Netconsole, which can be loaded from the kernel line in GRUB, compiled into your kernel, or loaded at runtime as a module. Having a netconsole entry in your menu.lst is most excellent for debugging slower computers like old laptops or thin-clients. It's easy to use. Just setup a 2nd computer (running arch) to accept syslog requests on a remote port, very fast and quick to do on arch-linux, 1 line to syslog.conf. Then you could use a log-color-parser like ccze to view all syslog logs, or just tail your everything.log. Then on your laptop, boot up and select the netconsole entry from the grub menu, and you will start seeing as much logging as you want on your syslog system. This logging lets you view even earlier log output than is available with the earlyprintk=vga kernel parameter, as netconsole is used by kernel hackers and developers, so it's very powerful.

title  Arch Linux DEBUG Netconsole
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro netconsole=514@10.0.0.2/12:34:56:78:9a:bc debug ignore_loglevel
initrd /kernel26.img

Hijacking cmdline

If you do not have access to GRUB or the kernel boottime cmdline, like on a server or virtual machine, as long as you have root permissions you can still enable this kind of simplistic verbose logging using a neat hack. While you cannot modify the /proc/cmdline even as root, you can place your own cmdline file on top of /proc/cmdline, so that accessing /proc/cmdline actually accesses your file.

For example if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo quiet

So I use a simple sed command to replace quiet with verbose like:

sed 's/ quiet/ verbose/' /proc/cmdline > /root/cmdline

Then I bind mount /root/cmdline so that it becomes /proc/cmdline, using the -n option to mount so that this mount won't be recorded in the systems mtab.

mount -n --bind -o ro /root/cmdline /proc/cmdline

Now if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo verbose

Advanced Linux Debugging using a Bootloader (GRUB) originally appeared on AskApache.com

I was trying to find an official, authoritative list of HTTP Status Codes but I kept finding lists that weren't authoritative or complete. So I searched and found my answer in the Apache HTTP Server source code. Once I had the exact HTTP Status Codes and resulting Error Documents sent by Apache, I researched deeper into HTTP Status Codes by reading as many related RFC's as I could find, and several other software source codes were explored. This is the most authoritative list I know of, if you can do better leave a comment and I'll update it. Another thing to keep in mind, the Status code number itself is what is used by software and hardware to make determinations, the phrase returned by the status code is for the human only and does not have any weight other than informing the user.. So "503 Service Unavailable", "503 Service Temporarily Unavailable", and "503 Get the heck outta here" are all completely valid.

Update March 9, 2009: A lot of sites on the web have updated their HTTP status code lists to include the HTTP Status codes listed on this page, including Wikipedia, IANA, W3C, and others, so rest assured this info is accurate and complete. If you'd like to see how to create custom error pages for all of these errors like mine /show-error-506 , then check out this detailed tutorial I just posted.

Contents

• List of All 57 HTTP Response Status Code
• Quick Start to triggering ErrorDocuments for each Status Code
• Automate the ErrorDocument Triggering
  • The htaccess Code
  • PHP script that gets and outputs the Headers/Content
• Headers and Content Returned
• Apache Source Code
  • httpd.h
  • http_protocol.c

57 APACHE HTTP STATUS RESPONSE CODES

Once I compiled the list of Apache recognized HTTP Status Codes, I was dying to see them all in action (i.e. the corresponding ErrorDocument). At first I thought I would have to create a php or perl script emulating each of the 57 HTTP Status Codes, a tedious undertaking I wasn't about to do. Instead I "asked Apache" by searching the Apache HTTP Documentation for ambiguity sending Status Codes and/or triggering ErrorDocuments with an Apache Directive.
While reading up on mod_alias and the Redirect directive I found:

Apache Docs

Syntax: Redirect [status] URL-path URL The status argument can be used to return other HTTP status codes. Other status codes can be returned by giving the numeric status code as the value of status. If the status is between 300 and 399, the URL argument must be present, otherwise it must be omitted.

100 Continue

ErrorDocument Continue | Sample 100 Continue
This means that the server has received the request headers, and that the client should proceed to send the request body (in case of a request which needs to be sent; for example, a POST request). If the request body is large, sending it to a server when a request has already been rejected based upon inappropriate headers is inefficient. To have a server check if the request could be accepted based on the requests headers alone, a client must send Expect: 100-continue as a header in its initial request (see RFC 2616 14.20 Expect header) and check if a 100 Continue status code is received in response before continuing (or receive 417 Expectation Failed and not continue).

101 Switching Protocols

ErrorDocument Switching Protocols | Sample 101 Switching Protocols
This means the requester has asked the server to switch protocols and the server is acknowledging that it will do so.[3]

102 Processing

ErrorDocument Processing | Sample 102 Processing
(WebDAV) - (RFC 2518 )

200 OK

ErrorDocument OK | Sample 200 OK
Standard response for successful HTTP requests. The actual response will depend on the request method used. In a GET request, the response will contain an entity corresponding to the requested resource. In a POST request the response will contain an entity describing or containing the result of the action.

201 Created

ErrorDocument Created | Sample 201 Created
The request has been fulfilled and resulted in a new resource being created.

202 Accepted

ErrorDocument Accepted | Sample 202 Accepted
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place.

203 Non-Authoritative Information

ErrorDocument Non-Authoritative Information | Sample 203 Non-Authoritative Information
The server successfully processed the request, but is returning information that may be from another source.

204 No Content

ErrorDocument No Content | Sample 204 No Content
The server successfully processed the request, but is not returning any content.

205 Reset Content

ErrorDocument Reset Content | Sample 205 Reset Content
The server successfully processed the request, but is not returning any content. Unlike a 204 response, this response requires that the requester reset the document view.

206 Partial Content

ErrorDocument Partial Content | Sample 206 Partial Content
The server is delivering only part of the resource due to a range header sent by the client. This is used by tools like wget to enable resuming of interrupted downloads, or split a download into multiple simultaneous streams.

207 Multi-Status

ErrorDocument Multi-Status | Sample 207 Multi-Status
(WebDAV) - The message body that follows is an XML message and can contain a number of separate response codes, depending on how many sub-requests were made.

226 IM Used

ErrorDocument IM Used | Sample 226 IM Used
The server has fulfilled a GET request for the resource, and the response is a representation of the result of one or more instance-manipulations applied to the current instance. The actual current instance might not be available except by combining this response with other previous or future responses, as appropriate for the specific instance-manipulation(s).

300 Multiple Choices

ErrorDocument Multiple Choices | Sample 300 Multiple Choices
Indicates multiple options for the resource that the client may follow. It, for instance, could be used to present different format options for video, list files with different extensions, or word sense disambiguation.

301 Moved Permanently

ErrorDocument Moved Permanently | Sample 301 Moved Permanently
This and all future requests should be directed to the given URI.

302 Found

ErrorDocument Found | Sample 302 Found
This is the most popular redirect code[citation needed], but also an example of industrial practice contradicting the standard. HTTP/1.0 specification (RFC 1945 ) required the client to perform a temporary redirect (the original describing phrase was "Moved Temporarily"), but popular browsers implemented it as a 303 See Other. Therefore, HTTP/1.1 added status codes 303 and 307 to disambiguate between the two behaviours. However, the majority of Web applications and frameworks still use the 302 status code as if it were the 303.

303 See Other

ErrorDocument See Other | Sample 303 See Other
The response to the request can be found under another URI using a GET method. When received in response to a PUT, it should be assumed that the server has received the data and the redirect should be issued with a separate GET message.

304 Not Modified

ErrorDocument Not Modified | Sample 304 Not Modified
Indicates the resource has not been modified since last requested. Typically, the HTTP client provides a header like the If-Modified-Since header to provide a time against which to compare. Utilizing this saves bandwidth and reprocessing on both the server and client, as only the header data must be sent and received in comparison to the entirety of the page being re-processed by the server, then resent using more bandwidth of the server and client.

305 Use Proxy

ErrorDocument Use Proxy | Sample 305 Use Proxy
Many HTTP clients (such as Mozilla[4] and Internet Explorer) do not correctly handle responses with this status code, primarily for security reasons.

306 Switch Proxy

ErrorDocument Switch Proxy | Sample 306 Switch Proxy
No longer used.

307 Temporary Redirect

ErrorDocument Temporary Redirect | Sample 307 Temporary Redirect
In this occasion, the request should be repeated with another URI, but future requests can still use the original URI. In contrast to 303, the request method should not be changed when reissuing the original request. For instance, a POST request must be repeated using another POST request.

400 Bad Request

ErrorDocument Bad Request | Sample 400 Bad Request
The request contains bad syntax or cannot be fulfilled.

401 Unauthorized

ErrorDocument Unauthorized | Sample 401 Unauthorized
Similar to 403 Forbidden, but specifically for use when authentication is possible but has failed or not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource. See Basic access authentication and Digest access authentication.

402 Payment Required

ErrorDocument Payment Required | Sample 402 Payment Required
The original intention was that this code might be used as part of some form of digital cash or micropayment scheme, but that has not happened, and this code has never been used.

403 Forbidden

ErrorDocument Forbidden | Sample 403 Forbidden
The request was a legal request, but the server is refusing to respond to it. Unlike a 401 Unauthorized response, authenticating will make no difference.

404 Not Found

ErrorDocument Not Found | Sample 404 Not Found
The requested resource could not be found but may be available again in the future. Subsequent requests by the client are permissible.

405 Method Not Allowed

ErrorDocument Method Not Allowed | Sample 405 Method Not Allowed
A request was made of a resource using a request method not supported by that resource; for example, using GET on a form which requires data to be presented via POST, or using PUT on a read-only resource.

406 Not Acceptable

ErrorDocument Not Acceptable | Sample 406 Not Acceptable
The requested resource is only capable of generating content not acceptable according to the Accept headers sent in the request.

407 Proxy Authentication Required

ErrorDocument Proxy Authentication Required | Sample 407 Proxy Authentication Required
Required

408 Request Timeout

ErrorDocument Request Timeout | Sample 408 Request Timeout
The server timed out waiting for the request.

409 Conflict

ErrorDocument Conflict | Sample 409 Conflict
Indicates that the request could not be processed because of conflict in the request, such as an edit conflict.

410 Gone

ErrorDocument Gone | Sample 410 Gone
Indicates that the resource requested is no longer available and will not be available again. This should be used when a resource has been intentionally removed; however, it is not necessary to return this code and a 404 Not Found can be issued instead. Upon receiving a 410 status code, the client should not request the resource again in the future. Clients such as search engines should remove the resource from their indexes.

411 Length Required

ErrorDocument Length Required | Sample 411 Length Required
The request did not specify the length of its content, which is required by the requested resource.

412 Precondition Failed

ErrorDocument Precondition Failed | Sample 412 Precondition Failed
The server does not meet one of the preconditions that the requester put on the request.

413 Request Entity Too Large

ErrorDocument Request Entity Too Large | Sample 413 Request Entity Too Large
The request is larger than the server is willing or able to process.

414 Request-URI Too Long

ErrorDocument Request-URI Too Long | Sample 414 Request-URI Too Long
The URI provided was too long for the server to process.

415 Unsupported Media Type

ErrorDocument Unsupported Media Type | Sample 415 Unsupported Media Type
The request did not specify any media types that the server or resource supports. For example the client specified that an image resource should be served as image/svg+xml, but the server cannot find a matching version of the image.

416 Requested Range Not Satisfiable

ErrorDocument Requested Range Not Satisfiable | Sample 416 Requested Range Not Satisfiable
The client has asked for a portion of the file, but the server cannot supply that portion (for example, if the client asked for a part of the file that lies beyond the end of the file).

417 Expectation Failed

ErrorDocument Expectation Failed | Sample 417 Expectation Failed
The server cannot meet the requirements of the Expect request-header field.

418 I'm a teapot

ErrorDocument I'm a teapot | Sample 418 I'm a teapot
The HTCPCP server is a teapot. The responding entity MAY be short and stout. Defined by the April Fools specification RFC 2324. See Hyper Text Coffee Pot Control Protocol for more information.

422 Unprocessable Entity

ErrorDocument Unprocessable Entity | Sample 422 Unprocessable Entity
(WebDAV) (RFC 4918 ) - The request was well-formed but was unable to be followed due to semantic errors.

423 Locked

ErrorDocument Locked | Sample 423 Locked
(WebDAV) (RFC 4918 ) - The resource that is being accessed is locked

424 Failed Dependency

ErrorDocument Failed Dependency | Sample 424 Failed Dependency
(WebDAV) (RFC 4918 ) - The request failed due to failure of a previous request (e.g. a PROPPATCH).

425 Unordered Collection

ErrorDocument Unordered Collection | Sample 425 Unordered Collection
Defined in drafts of WebDav Advanced Collections, but not present in "Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol" (RFC 3648).

426 Upgrade Required

ErrorDocument Upgrade Required | Sample 426 Upgrade Required
(RFC 2817 ) - The client should switch to TLS/1.0.

449 Retry With

ErrorDocument Retry With | Sample 449 Retry With
A Microsoft extension. The request should be retried after doing the appropriate action.

500 Internal Server Error

ErrorDocument Internal Server Error | Sample 500 Internal Server Error
A generic error message, given when no more specific message is suitable.

501 Not Implemented

ErrorDocument Not Implemented | Sample 501 Not Implemented
The server either does not recognise the request method, or it lacks the ability to fulfil the request.

502 Bad Gateway

ErrorDocument Bad Gateway | Sample 502 Bad Gateway
The server was acting as a gateway or proxy and received an invalid response from the upstream server.

503 Service Unavailable

ErrorDocument Service Unavailable | Sample 503 Service Unavailable
The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

504 Gateway Timeout

ErrorDocument Gateway Timeout | Sample 504 Gateway Timeout
The server was acting as a gateway or proxy and did not receive a timely request from the upstream server.

505 HTTP Version Not Supported

ErrorDocument HTTP Version Not Supported | Sample 505 HTTP Version Not Supported
The server does not support the HTTP protocol version used in the request.

506 Variant Also Negotiates

ErrorDocument Variant Also Negotiates | Sample 506 Variant Also Negotiates
(RFC 2295 ) - Transparent content negotiation for the request, results in a circular reference.

507 Insufficient Storage

ErrorDocument Insufficient Storage | Sample 507 Insufficient Storage
(WebDAV) (RFC 4918 )

509 Bandwidth Limit Exceeded

ErrorDocument Bandwidth Limit Exceeded | Sample 509 Bandwidth Limit Exceeded
(Apache bw/limited extension) - This status code, while used by many servers, is not specified in any RFCs.

510 Not Extended

ErrorDocument Not Extended | Sample 510 Not Extended
(RFC 2774 ) - Further extensions to the request are required for the server to fulfil it.

1xx Info / Informational

HTTP_INFO - Request received, continuing process. Indicates a provisional response, consisting only of the Status-Line and optional headers, and is terminated by an empty line.

• 100 Continue - HTTP_CONTINUE
• 101 Switching Protocols - HTTP_SWITCHING_PROTOCOLS
• 102 Processing - HTTP_PROCESSING

2xx Success / OK

HTTP_SUCCESS - The action was successfully received, understood, and accepted. Indicates that the client's request was successfully received, understood, and accepted.

• 200 OK - HTTP_OK
• 201 Created - HTTP_CREATED
• 202 Accepted - HTTP_ACCEPTED
• 203 Non-Authoritative Information - HTTP_NON_AUTHORITATIVE
• 204 No Content - HTTP_NO_CONTENT
• 205 Reset Content - HTTP_RESET_CONTENT
• 206 Partial Content - HTTP_PARTIAL_CONTENT
• 207 Multi-Status - HTTP_MULTI_STATUS

3xx Redirect

HTTP_REDIRECT - The client must take additional action to complete the request. Indicates that further action needs to be taken by the user-agent in order to fulfill the request. The action required may be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD. A user agent should not automatically redirect a request more than 5 times, since such redirections usually indicate an infinite loop.

• 300 Multiple Choices - HTTP_MULTIPLE_CHOICES
• 301 Moved Permanently - HTTP_MOVED_PERMANENTLY
• 302 Found - HTTP_MOVED_TEMPORARILY
• 303 See Other - HTTP_SEE_OTHER
• 304 Not Modified - HTTP_NOT_MODIFIED
• 305 Use Proxy - HTTP_USE_PROXY
• 306 unused - UNUSED
• 307 Temporary Redirect - HTTP_TEMPORARY_REDIRECT

4xx Client Error

HTTP_CLIENT_ERROR - The request contains bad syntax or cannot be fulfilled. Indicates case where client seems to have erred. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition.

• 400 Bad Request - HTTP_BAD_REQUEST
• 401 Authorization Required - HTTP_UNAUTHORIZED
• 402 Payment Required - HTTP_PAYMENT_REQUIRED
• 403 Forbidden - HTTP_FORBIDDEN
• 404 Not Found - HTTP_NOT_FOUND
• 405 Method Not Allowed - HTTP_METHOD_NOT_ALLOWED
• 406 Not Acceptable - HTTP_NOT_ACCEPTABLE
• 407 Proxy Authentication Required - HTTP_PROXY_AUTHENTICATION_REQUIRED
• 408 Request Time-out - HTTP_REQUEST_TIME_OUT
• 409 Conflict - HTTP_CONFLICT
• 410 Gone - HTTP_GONE
• 411 Length Required - HTTP_LENGTH_REQUIRED
• 412 Precondition Failed - HTTP_PRECONDITION_FAILED
• 413 Request Entity Too Large - HTTP_REQUEST_ENTITY_TOO_LARGE
• 414 Request-URI Too Large - HTTP_REQUEST_URI_TOO_LARGE
• 415 Unsupported Media Type - HTTP_UNSUPPORTED_MEDIA_TYPE
• 416 Requested Range Not Satisfiable - HTTP_RANGE_NOT_SATISFIABLE
• 417 Expectation Failed - HTTP_EXPECTATION_FAILED
• 418 I'm a teapot - UNUSED
• 419 unused - UNUSED
• 420 unused - UNUSED
• 421 unused - UNUSED
• 422 Unprocessable Entity - HTTP_UNPROCESSABLE_ENTITY
• 423 Locked - HTTP_LOCKED
• 424 Failed Dependency - HTTP_FAILED_DEPENDENCY
• 425 No code - HTTP_NO_CODE
• 426 Upgrade Required - HTTP_UPGRADE_REQUIRED

5xx Server Error

HTTP_SERVER_ERROR - The server failed to fulfill an apparently valid request. Indicate cases in which the server is aware that it has erred or is incapable of performing the request. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. User agents should display any included entity to the user. These response codes are applicable to any request method.

• 500 Internal Server Error - HTTP_INTERNAL_SERVER_ERROR
• 501 Method Not Implemented - HTTP_NOT_IMPLEMENTED
• 502 Bad Gateway - HTTP_BAD_GATEWAY
• 503 Service Temporarily Unavailable - HTTP_SERVICE_UNAVAILABLE
• 504 Gateway Time-out - HTTP_GATEWAY_TIME_OUT
• 505 HTTP Version Not Supported - HTTP_VERSION_NOT_SUPPORTED
• 506 Variant Also Negotiates - HTTP_VARIANT_ALSO_VARIES
• 507 Insufficient Storage - HTTP_INSUFFICIENT_STORAGE
• 508 unused - UNUSED
• 509 unused - UNUSED
• 510 Not Extended - HTTP_NOT_EXTENDED

Quick Start to triggering ErrorDocuments for each Status Code

Let start with a quick and easy example. Add the following Redirect rules to your htaccess file, then open your browser and goto each url like yoursite.com/e/400. Don't create an /e/ directory or any files.

Redirect 400 /e/400
Redirect 503 /e/503
Redirect 405 /e/405

Automate the ErrorDocument Triggering

The htaccess Redirects

When a Status code is encountered, Apache outputs the Header and the ErrorDocument for that error code. So you can view any Header and the default ErrorDocument, by causing that numerical error code, which is caused by the Status Code.

For instance, if you request a file that doesn't exist, a 404 Not Found Header is issued and the corresponding ErrorDocument is served with the 404 Not Found Header.

Redirect 100 /e/100
Redirect 101 /e/101
Redirect 102 /e/102
Redirect 200 /e/200
Redirect 201 /e/201
Redirect 202 /e/202
Redirect 203 /e/203
Redirect 204 /e/204
Redirect 205 /e/205
Redirect 206 /e/206
Redirect 207 /e/207
Redirect 300 /e/300 http://www.askapache.com/?s=300
Redirect 301 /e/301 http://www.askapache.com/?s=301
Redirect 302 /e/302 http://www.askapache.com/?s=302
Redirect 303 /e/303 http://www.askapache.com/?s=303
Redirect 304 /e/304 http://www.askapache.com/?s=304
Redirect 305 /e/305 http://www.askapache.com/?s=305
Redirect 306 /e/306 http://www.askapache.com/?s=306
Redirect 307 /e/307 http://www.askapache.com/?s=307
Redirect 400 /e/400
Redirect 401 /e/401
Redirect 402 /e/402
Redirect 403 /e/403
Redirect 404 /e/404
Redirect 405 /e/405
Redirect 406 /e/406
Redirect 407 /e/407
Redirect 408 /e/408
Redirect 409 /e/409
Redirect 410 /e/410
Redirect 411 /e/411
Redirect 412 /e/412
Redirect 413 /e/413
Redirect 414 /e/414
Redirect 415 /e/415
Redirect 416 /e/416
Redirect 417 /e/417
Redirect 418 /e/418
Redirect 419 /e/419
Redirect 420 /e/420
Redirect 421 /e/421
Redirect 422 /e/422
Redirect 423 /e/423
Redirect 424 /e/424
Redirect 425 /e/425
Redirect 426 /e/426
Redirect 500 /e/500
Redirect 501 /e/501
Redirect 502 /e/502
Redirect 503 /e/503
Redirect 504 /e/504
Redirect 505 /e/505
Redirect 506 /e/506
Redirect 507 /e/507
Redirect 508 /e/508
Redirect 509 /e/509
Redirect 510 /e/510

PHP script that gets and outputs the Headers/Content

Now all I have to do is add 57 Redirect Directives to my htaccess, and then request each of them 1 at a time from my browser to see the result, and use a packet sniffing program like WireShark to see the Headers. Uh, scratch that, that would take way too long!

Instead I hacked up a simple php script using cURL to automate sending GET Requests to each of the 57 Redirect URL-paths. A side benefit of using the php script is that it performs all 57 Requests concurrently and saves each Requests returned headers and content to an output buffer. After all 57 have been queried, the output buffer is flushed to the browser.

<?php
$SITENAME='http://www.askapache.com';
 
$CODES = array(array('100','101','102'),
array('200','201','202','203','204','205','206','207'),
array('300','301','302','303','304','305','306','307'),
array('400','401','402','403','404','405','406','407','408','409','410','411','412','413',
'414','415','416','417','418','419','420','421','422','423','424','425','426'),
array('500','501','502','503','504','505','506','507','508','509','510'));
 
$TMPSAVETO='/tmp/'.time().'.txt';
 
# if file exists then delete it
if(is_file($TMPSAVETO))unlink($TMPSAVETO);
 
foreach($CODES as $keyd => $res)
{
foreach($res as $key)
{
$ch = curl_init("$SITENAME/e/$key");
$fp = fopen ($TMPSAVETO, "a");
curl_setopt ($ch, CURLOPT_FILE, $fp);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION ,1);
curl_setopt ($ch, CURLOPT_HEADER ,1);
curl_exec ($ch);
curl_close ($ch);
fclose ($fp);
}
}
$OUT='';
ob_start();
header ("Content-Type: text/plain;");
readfile($TMPSAVETO);
$OUT=ob_get_clean();
echo $OUT;
unlink($TMPSAVETO);
exit;
?>

Headers and Content Returned

100 Continue

HTTP/1.1 100 Continue
<html>
<head>
<title>100 Continue</title>
</head>
<body>
<h1>Continue</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

101 Switching Protocols

HTTP/1.1 101 Switching Protocols<html>
<head>
<title>101 Switching Protocols</title>
</head>
<body>
<h1>Switching Protocols</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

102 Processing

HTTP/1.1 102 Processing
X-Pad: avoid browser bug<html>
<head>
<title>102 Processing</title>
</head>
<body>
<h1>Processing</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

200 OK

HTTP/1.1 200 OK
<html>
<head>
<title>200 OK</title>
</head>
<body>
<h1>OK</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

201 Created

HTTP/1.1 201 Created
<html>
<head>
<title>201 Created</title>
</head>
<body>
<h1>Created</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

202 Accepted

HTTP/1.1 202 Accepted
<html>
<head>
<title>202 Accepted</title>
</head>
<body>
<h1>Accepted</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

203 Non-Authoritative Information

HTTP/1.1 203 Non-Authoritative Information
<html>
<head>
<title>203 Non-Authoritative Information</title>
</head>
<body>
<h1>Non-Authoritative Information</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

204 No Content

HTTP/1.1 204 No Content
Content-Type: text/plain; charset=UTF-8

205 Reset Content

HTTP/1.1 205 Reset Content<html>
<head>
<title>205 Reset Content</title>
</head>
<body>
<h1>Reset Content</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

206 Partial Content

HTTP/1.1 206 Partial Content<html>
<head>
<title>206 Partial Content</title>
</head>
<body>
<h1>Partial Content</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

207 Multi-Status

HTTP/1.1 207 Multi-Status
X-Pad: avoid browser bug<html>
<head>
<title>207 Multi-Status</title>
</head>
<body>
<h1>Multi-Status</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

300 Multiple Choices

HTTP/1.1 300 Multiple Choices
Location: http://www.askapache.com/?s=300<html>
<head>
<title>300 Multiple Choices</title>
</head>
<body>
<h1>Multiple Choices</h1>
</body>
</html>

301 Moved Permanently

HTTP/1.1 301 Moved Permanently
Location: http://www.askapache.com/?s=301<html>
<head>
<title>301 Moved Permanently</title>
</head>
<body>
<h1>Moved Permanently</h1>
<p>The document has moved  <a href="http://www.askapache.com/?s=301">here</a> .</p>
</body>
</html>

302 Found

HTTP/1.1 302 Found
Location: http://www.askapache.com/?s=302<html>
<head>
<title>302 Found</title>
</head>
<body>
<h1>Found</h1>
<p>The document has moved  <a href="http://www.askapache.com/?s=302">here</a> .</p>
</body>
</html>

303 See Other

HTTP/1.1 303 See Other
Location: http://www.askapache.com/?s=303<html>
<head>
<title>303 See Other</title>
</head>
<body>
<h1>See Other</h1>
<p>The answer to your request is located  <a href="http://www.askapache.com/?s=303">here</a> .</p>
</body>
</html>

304 Not Modified

HTTP/1.1 304 Not Modified

305 Use Proxy

HTTP/1.1 305 Use Proxy
Location: http://www.askapache.com/?s=305<html>
<head>
<title>305 Use Proxy</title>
</head>
<body>
<h1>Use Proxy</h1>
<p>This resource is only accessible through the proxy
    http://www.askapache.com/?s=305<br />You will need to configure your client to use that proxy.</p>
</body>
</html>

306 unused

HTTP/1.1 306 unused
Location: http://www.askapache.com/?s=306<html>
<head>
<title>306 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

307 Temporary Redirect

HTTP/1.1 307 Temporary Redirect
Location: http://www.askapache.com/?s=307<html>
<head>
<title>307 Temporary Redirect</title>
</head>
<body>
<h1>Temporary Redirect</h1>
<p>The document has moved  <a href="http://www.askapache.com/?s=307">here</a> .</p>
</body>
</html>

400 Bad Request

HTTP/1.1 400 Bad Request
Connection: close<html>
<head>
<title>400 Bad Request</title>
</head>
<body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.</p>
</body>
</html>

401 Authorization Required

HTTP/1.1 401 Authorization Required<html>
<head>
<title>401 Authorization Required</title>
</head>
<body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
    are authorized to access the document
    requested.  Either you supplied the wrong
    credentials (e.g., bad password), or your
    browser doesn't understand how to supply
    the credentials required.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body>
</html>

402 Payment Required

HTTP/1.1 402 Payment Required<html>
<head>
<title>402 Payment Required</title>
</head>
<body>
<h1>Payment Required</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

403 Forbidden

HTTP/1.1 403 Forbidden<html>
<head>
<title>403 Forbidden</title>
</head>
<body>
<h1>Forbidden</h1>
<p>You don't have permission to access /e/403
    on this server.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body>
</html>

404 Not Found

HTTP/1.1 404 Not Found<html>
<head>
<title>404 Not Found</title>
</head>
<body>
<h1>Not Found</h1>
<p>The requested URL /e/404 was not found on this server.</p>
</body>
</html>

NOTE:
You will most definately want to check out and use the Google 404 Error Page if you run WordPress.

405 Method Not Allowed

HTTP/1.1 405 Method Not Allowed
Allow: TRACE
<html>
<head>
<title>405 Method Not Allowed</title>
</head>
<body>
<h1>Method Not Allowed</h1>
<p>The requested method GET is not allowed for the URL /e/405.</p>
</body>
</html>

406 Not Acceptable

HTTP/1.1 406 Not Acceptable
<html>
<head>
<title>406 Not Acceptable</title>
</head>
<body>
<h1>Not Acceptable</h1>
<p>An appropriate representation of the requested resource /e/406 could not be found on this server.</p>
</body>
</html>

407 Proxy Authentication Required

HTTP/1.1 407 Proxy Authentication Required<html>
<head>
<title>407 Proxy Authentication Required</title>
</head>
<body>
<h1>Proxy Authentication Required</h1>
<p>This server could not verify that you
    are authorized to access the document
    requested.  Either you supplied the wrong
    credentials (e.g., bad password), or your
    browser doesn't understand how to supply
    the credentials required.</p>
</body>
</html>

408 Request Time-out

HTTP/1.1 408 Request Time-out
Connection: close
<html>
<head>
<title>408 Request Time-out</title>
</head>
<body>
<h1>Request Time-out</h1>
<p>Server timeout waiting for the HTTP request from the client.</p>
</body>
</html>

409 Conflict

HTTP/1.1 409 Conflict
<html>
<head>
<title>409 Conflict</title>
</head>
<body>
<h1>Conflict</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

410 Gone

HTTP/1.1 410 Gone
<html>
<head>
<title>410 Gone</title>
</head>
<body>
<h1>Gone</h1>
<p>The requested resource<br />/e/410<br />is no longer available on this server and there is no forwarding address.
    Please remove all references to this resource.</p>
</body>
</html>

411 Length Required

HTTP/1.1 411 Length Required
Connection: close
<html>
<head>
<title>411 Length Required</title>
</head>
<body>
<h1>Length Required</h1>
<p>A request of the requested method GET requires a valid Content-length.</p>
</body>
</html>

412 Precondition Failed

HTTP/1.1 412 Precondition Failed
<html>
<head>
<title>412 Precondition Failed</title>
</head>
<body>
<h1>Precondition Failed</h1>
<p>The precondition on the request for the URL /e/412 evaluated to false.</p>
</body>
</html>

413 Request Entity Too Large

HTTP/1.1 413 Request Entity Too Large
Connection: close<html>
<head>
<title>413 Request Entity Too Large</title>
</head>
<body>
<h1>Request Entity Too Large</h1>
The requested resource<br />/e/413<br />does not allow request data with GET requests, or the amount of data provided in
the request exceeds the capacity limit.
</body>
</html>

414 Request-URI Too Large

HTTP/1.1 414 Request-URI Too Large
Connection: close<html>
<head>
<title>414 Request-URI Too Large</title>
</head>
<body>
<h1>Request-URI Too Large</h1>
<p>The requested URL's length exceeds the capacity
    limit for this server.</p>
</body>
</html>

415 Unsupported Media Type

HTTP/1.1 415 Unsupported Media Type<html>
<head>
<title>415 Unsupported Media Type</title>
</head>
<body>
<h1>Unsupported Media Type</h1>
<p>The supplied request data is not in a format
    acceptable for processing by this resource.</p>
</body>
</html>

416 Requested Range Not Satisfiable

HTTP/1.1 416 Requested Range Not Satisfiable

417 Expectation Failed

HTTP/1.1 417 Expectation Failed<html>
<head>
<title>417 Expectation Failed</title>
</head>
<body>
<h1>Expectation Failed</h1>
<p>The expectation given in the Expect request-header
    field could not be met by this server.</p>
<p>The client sent<pre>
    Expect: </p>
</body>
</html>

418 I'm a teapot

HTTP/1.1 418 I'm a teapot<html>
<head>
<title>418 I'm a teapot</title>
</head>
<body>
<h1>I'm a teapot</h1>
<p>Unfortunately this coffee machine is out of coffee.</p>
</body>
</html>

419 unused

HTTP/1.1 419 unused<html>
<head>
<title>419 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

420 unused

HTTP/1.1 420 unused<html>
<head>
<title>420 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

421 unused

HTTP/1.1 421 unused<html>
<head>
<title>421 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

422 Unprocessable Entity

HTTP/1.1 422 Unprocessable Entity<html>
<head>
<title>422 Unprocessable Entity</title>
</head>
<body>
<h1>Unprocessable Entity</h1>
<p>The server understands the media type of the
    request entity, but was unable to process the
    contained instructions.</p>
</body>
</html>

423 Locked

HTTP/1.1 423 Locked<html>
<head>
<title>423 Locked</title>
</head>
<body>
<h1>Locked</h1>
<p>The requested resource is currently locked.
    The lock must be released or proper identification
    given before the method can be applied.</p>
</body>
</html>

424 Failed Dependency

HTTP/1.1 424 Failed Dependency<html>
<head>
<title>424 Failed Dependency</title>
</head>
<body>
<h1>Failed Dependency</h1>
<p>The method could not be performed on the resource
    because the requested action depended on another
    action and that other action failed.</p>
</body>
</html>

425 No code

HTTP/1.1 425 No code<html>
<head>
<title>425 No code</title>
</head>
<body>
<h1>No code</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

426 Upgrade Required

HTTP/1.1 426 Upgrade Required<html>
<head>
<title>426 Upgrade Required</title>
</head>
<body>
<h1>Upgrade Required</h1>
<p>The requested resource can only be retrieved
    using SSL.  The server is willing to upgrade the current
    connection to SSL, but your client doesn't support it.
    Either upgrade your client, or try requesting the page
    using https:// </p>
</body>
</html>

500 Internal Server Error

HTTP/1.1 500 Internal Server Error
Connection: close<html>
<head>
<title>500 Internal Server Error</title>
</head>
<body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body>
</html>

501 Method Not Implemented

HTTP/1.1 501 Method Not Implemented
Allow: TRACE
Connection: close<html>
<head>
<title>501 Method Not Implemented</title>
</head>
<body>
<h1>Method Not Implemented</h1>
<p>GET to /e/501 not supported.</p>
</body>
</html>

502 Bad Gateway

HTTP/1.1 502 Bad Gateway
X-Pad: avoid browser bug<html>
<head>
<title>502 Bad Gateway</title>
</head>
<body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
    response from an upstream server.</p>
</body>
</html>

503 Service Temporarily Unavailable

HTTP/1.1 503 Service Temporarily Unavailable
Connection: close<html>
<head>
<title>503 Service Temporarily Unavailable</title>
</head>
<body>
<h1>Service Temporarily Unavailable</h1>
<p>The server is temporarily unable to service your
    request due to maintenance downtime or capacity
    problems. Please try again later.</p>
</body>
</html>

504 Gateway Time-out

HTTP/1.1 504 Gateway Time-out<html>
<head>
<title>504 Gateway Time-out</title>
</head>
<body>
<h1>Gateway Time-out</h1>
<p>The proxy server did not receive a timely response
    from the upstream server.</p>
</body>
</html>

505 HTTP Version Not Supported

HTTP/1.1 505 HTTP Version Not Supported<html>
<head>
<title>505 HTTP Version Not Supported</title>
</head>
<body>
<h1>HTTP Version Not Supported</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

506 Variant Also Negotiates

HTTP/1.1 506 Variant Also Negotiates<html>
<head>
<title>506 Variant Also Negotiates</title>
</head>
<body>
<h1>Variant Also Negotiates</h1>
<p>A variant for the requested resource
    <pre>
    /e/506
    </pre>
    is itself a negotiable resource. This indicates a configuration error.</p>
</body>
</html>

507 Insufficient Storage

HTTP/1.1 507 Insufficient Storage<html>
<head>
<title>507 Insufficient Storage</title>
</head>
<body>
<h1>Insufficient Storage</h1>
<p>The method could not be performed on the resource
    because the server is unable to store the
    representation needed to successfully complete the
    request.  There is insufficient free space left in
    your storage allocation.</p>
</body>
</html>

508 unused

HTTP/1.1 508 unused<html>
<head>
<title>508 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

509 unused

HTTP/1.1 509 unused
<html>
<head>
<title>509 unused</title>
</head>
<body>
<h1>unused</h1>
<p>The server encountered an internal error or misconfigurationand was unable to complete your request.</p>
<p>Please contact the server administrator, a@s.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p>
<p>More information about this error may be available in the server error log.</p>
</body>
</html>

510 Not Extended

HTTP/1.1 510 Not Extended
X-Pad: avoid browser bug
<html>
<head>
<title>510 Not Extended</title>
</head>
<body>
<h1>Not Extended</h1>
<p>A mandatory extension policy in the request is not
    accepted by the server for this resource.</p>
</body>
</html>

Apache Source Code

httpd.h

From httpd.h

/**
* The size of the static array in http_protocol.c for storing
* all of the potential response status-lines (a sparse table).
* A future version should dynamically generate the apr_table_t at startup.
*/
#define RESPONSE_CODES 57
#define HTTP_CONTINUE            100
#define HTTP_SWITCHING_PROTOCOLS       101
#define HTTP_PROCESSING          102
#define HTTP_OK              200
#define HTTP_CREATED             201
#define HTTP_ACCEPTED            202
#define HTTP_NON_AUTHORITATIVE       203
#define HTTP_NO_CONTENT          204
#define HTTP_RESET_CONTENT         205
#define HTTP_PARTIAL_CONTENT         206
#define HTTP_MULTI_STATUS          207
#define HTTP_MULTIPLE_CHOICES        300
#define HTTP_MOVED_PERMANENTLY       301
 
#define HTTP_MOVED_TEMPORARILY       302
#define HTTP_SEE_OTHER           303
#define HTTP_NOT_MODIFIED          304
#define HTTP_USE_PROXY           305
#define HTTP_TEMPORARY_REDIRECT      307
#define HTTP_BAD_REQUEST           400
#define HTTP_UNAUTHORIZED          401
#define HTTP_PAYMENT_REQUIRED        402
#define HTTP_FORBIDDEN           403
#define HTTP_NOT_FOUND           404
#define HTTP_METHOD_NOT_ALLOWED      405
#define HTTP_NOT_ACCEPTABLE        406
#define HTTP_PROXY_AUTHENTICATION_REQUIRED 407
#define HTTP_REQUEST_TIME_OUT        408
#define HTTP_CONFLICT            409
#define HTTP_GONE              410
#define HTTP_LENGTH_REQUIRED         411
#define HTTP_PRECONDITION_FAILED       412
#define HTTP_REQUEST_ENTITY_TOO_LARGE    413
#define HTTP_REQUEST_URI_TOO_LARGE     414
#define HTTP_UNSUPPORTED_MEDIA_TYPE    415
#define HTTP_RANGE_NOT_SATISFIABLE     416
#define HTTP_EXPECTATION_FAILED      417
#define HTTP_UNPROCESSABLE_ENTITY      422
#define HTTP_LOCKED            423
#define HTTP_FAILED_DEPENDENCY       424
#define HTTP_UPGRADE_REQUIRED        426
#define HTTP_INTERNAL_SERVER_ERROR     500
#define HTTP_NOT_IMPLEMENTED         501
#define HTTP_BAD_GATEWAY           502
#define HTTP_SERVICE_UNAVAILABLE       503
#define HTTP_GATEWAY_TIME_OUT        504
#define HTTP_VERSION_NOT_SUPPORTED     505
#define HTTP_VARIANT_ALSO_VARIES       506
#define HTTP_INSUFFICIENT_STORAGE      507
#define HTTP_NOT_EXTENDED          510
 
/** is the status code informational */
#define ap_is_HTTP_INFO(x)     (((x) >= 100)&&((x) < 200))
/** is the status code OK ?*/
#define ap_is_HTTP_SUCCESS(x)    (((x) >= 200)&&((x) < 300))
/** is the status code a redirect */
#define ap_is_HTTP_REDIRECT(x)   (((x) >= 300)&&((x) < 400))
/** is the status code a error (client or server) */
#define ap_is_HTTP_ERROR(x)    (((x) >= 400)&&((x) < 600))
/** is the status code a client error  */
#define ap_is_HTTP_CLIENT_ERROR(x) (((x) >= 400)&&((x) < 500))
/** is the status code a server error  */
#define ap_is_HTTP_SERVER_ERROR(x) (((x) >= 500)&&((x) < 600))
/** is the status code a (potentially) valid response code?  */
#define ap_is_HTTP_VALID_RESPONSE(x) (((x) >= 100)&&((x) < 600))
 
/** should the status code drop the connection */
#define ap_status_drops_connection(x) \
(((x) == HTTP_BAD_REQUEST)       || \
((x) == HTTP_REQUEST_TIME_OUT)    || \
((x) == HTTP_LENGTH_REQUIRED)     || \
((x) == HTTP_REQUEST_ENTITY_TOO_LARGE) || \
((x) == HTTP_REQUEST_URI_TOO_LARGE) || \
((x) == HTTP_INTERNAL_SERVER_ERROR) || \
((x) == HTTP_SERVICE_UNAVAILABLE) || \
((x) == HTTP_NOT_IMPLEMENTED))

HTTP_INFO

Is the status code (x) informational?

x >= 100 && x < 200

HTTP_SUCCESS

Is the status code (x) OK?

x >= 200 && x < 300

HTTP_REDIRECT

Is the status code (x) a redirect?

x >= 300 && x < 400

HTTP_ERROR

Is the status code (x) a error (client or server)?

x >= 400 && x < 600

HTTP_CLIENT_ERROR

Is the status code (x) a client error?

x >= 400 && x < 500

HTTP_SERVER_ERROR

Is the status code (x) a server error?

x >= 500 && x < 600

HTTP_VALID_RESPONSE

Is the status code (x) a (potentially) valid response code?

x >= 100 && x < 600

http_protocol.c

From http_protocol.c

static const char * status_lines[RESPONSE_CODES] =
static const char * const status_lines[RESPONSE_CODES] =
"100 Continue",
"101 Switching Protocols",
"102 Processing",
"200 OK",
"201 Created",
"202 Accepted",
"203 Non-Authoritative Information",
"204 No Content",
"205 Reset Content",
"206 Partial Content",
"207 Multi-Status",
"300 Multiple Choices",
"301 Moved Permanently",
"302 Found",
"303 See Other",
"304 Not Modified",
"305 Use Proxy",
"306 unused",
"307 Temporary Redirect",
"400 Bad Request",
"401 Authorization Required",
"402 Payment Required",
"403 Forbidden",
"404 Not Found",
"405 Method Not Allowed",
"406 Not Acceptable",
"407 Proxy Authentication Required",
"408 Request Time-out",
"409 Conflict",
"410 Gone",
"411 Length Required",
"412 Precondition Failed",
"413 Request Entity Too Large",
"414 Request-URI Too Large",
"415 Unsupported Media Type",
"416 Requested Range Not Satisfiable",
"417 Expectation Failed",
"418 unused",
"419 unused",
"420 unused",
"421 unused",
"422 Unprocessable Entity",
"423 Locked",
"424 Failed Dependency",
"425 No code",
"426 Upgrade Required",
"500 Internal Server Error",
"501 Method Not Implemented",
"502 Bad Gateway",
"503 Service Temporarily Unavailable",
"504 Gateway Time-out",
"505 HTTP Version Not Supported",
"506 Variant Also Negotiates",
"507 Insufficient Storage",
"508 unused",
"509 unused",
"510 Not Extended"

IANA HTTP Status Code Registry

WordPress 2.8 Changes

I just learned that my modification to the WordPress core was accepted and will be implemented for version 2.8! This may mean WordPress is the only 100% HTTP/1.1 compliant software on the net! Below is the new list (I thought someone out there could use the php array) and as you can see, unfortunately418 I'm a teapotdidn't make it ;)

$wp_header_to_desc = array(
  100 => 'Continue',
  101 => 'Switching Protocols',
  102 => 'Processing',
 
  200 => 'OK',
  201 => 'Created',
  202 => 'Accepted',
  203 => 'Non-Authoritative Information',
  204 => 'No Content',
  205 => 'Reset Content',
  206 => 'Partial Content',
  207 => 'Multi-Status',
  226 => 'IM Used',
 
  300 => 'Multiple Choices',
  301 => 'Moved Permanently',
  302 => 'Found',
  303 => 'See Other',
  304 => 'Not Modified',
  305 => 'Use Proxy',
  306 => 'Reserved',
  307 => 'Temporary Redirect',
 
  400 => 'Bad Request',
  401 => 'Unauthorized',
  402 => 'Payment Required',
  403 => 'Forbidden',
  404 => 'Not Found',
  405 => 'Method Not Allowed',
  406 => 'Not Acceptable',
  407 => 'Proxy Authentication Required',
  408 => 'Request Timeout',
  409 => 'Conflict',
  410 => 'Gone',
  411 => 'Length Required',
  412 => 'Precondition Failed',
  413 => 'Request Entity Too Large',
  414 => 'Request-URI Too Long',
  415 => 'Unsupported Media Type',
  416 => 'Requested Range Not Satisfiable',
  417 => 'Expectation Failed',
  422 => 'Unprocessable Entity',
  423 => 'Locked',
  424 => 'Failed Dependency',
  426 => 'Upgrade Required',
 
  500 => 'Internal Server Error',
  501 => 'Not Implemented',
  502 => 'Bad Gateway',
  503 => 'Service Unavailable',
  504 => 'Gateway Timeout',
  505 => 'HTTP Version Not Supported',
  506 => 'Variant Also Negotiates',
  507 => 'Insufficient Storage',
  510 => 'Not Extended'
);

RIPE WHOIS

All the status codes are standard HTTP codes ( http://www.iana.org/assignments/http-status-codes ).

Clients should avoid any form of coupling with the the text/plain error message contained in response body since it may change between different releases of the API and is only intended as a starting point for indentifying the real causes of the exception event.

The following table gives a brief description of the mapping between standard Whois V.3 responses and the related REST services status codes. Consider this table as just an example of the error mapping strategy, it may change with future releases.

Clients will have to define error messages generic enough to represent the four main error conditions, that are Bad Request, Forbidden, Not Found and Internal Server Error.

For example a possible mapping for client side error messages may be:

Helpful HTTP Links

1. IANA registry
2. Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)
3. Adobe Flash status code definitions (ie 408)
4. Microsoft Internet Information Server Status Codes and Sub-Codes
5. httplint
6. HTTP Headers, brief intro.
7. Common User-Agent Issues
8. [RFC2295] Holtman, K. and A.H. Mutz , " Transparent Content Negotiation in HTTP ", RFC 2295, March 1998.
9. [RFC2518] Goland, Y. , Whitehead, E. , Faizi, A. , Carter, S.R. , and D. Jensen , " HTTP Extensions for Distributed Authoring -- WEBDAV ", RFC 2518, February 1999.
10. [RFC2616] Fielding, R. , Gettys, J. , Mogul, J. , Frystyk, H. , Masinter, L. , Leach, P. , and T. Berners-Lee , " Hypertext Transfer Protocol -- HTTP/1.1 ", RFC 2616, June 1999.
11. [RFC2774] Nielsen, H. , Leach, P. , and S. Lawrence , " An HTTP Extension Framework ", RFC 2774, February 2000.
12. [RFC2817] Khare, R. and S. Lawrence, " Upgrading to TLS Within HTTP/1.1 ", RFC 2817, May 2000.
13. [RFC3229] Mogul, J., Krishnamurthy, B., Douglis, F., Feldmann, A., Goland, Y., van Hoff, A., and D. Hellerstein, " Delta encoding in HTTP ", RFC 3229, January 2002.
14. [RFC4918] Dusseault, L., Ed. , " HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV) ", RFC 4918, June 2007.

HTTP Status Codes and Htaccess ErrorDocuments originally appeared on AskApache.com

SetEnvIf and SetEnvIfNoCase are really useful directives supplied by the mod_setenvif module that allow you to conditionally set environment variables accessible by scripts and apache based on the value of HTTP Headers, Other Variables, and Request information.

For debugging, you may want to use my server environment variable debugging script

• Unique mod_setenvif Variables
• Populates HTTP_MY_ Variables with mod_setenvif variable values
• Set REMOTE_HOST to HTTP_HOST
• Allows only if HOST Header is present in request
• Add values from HTTP Headers
• Set the REDIRECT_STATUS for Interpreter Security

Unique mod_setenvif Variables

These can be used for attribute.

Remote_Host

the hostname (if available) of the client making the request - crawl-66-249-70-24.googlebot.com

Remote_Addr

IP address of the client making the request - 66.249.70.24

Server_Addr

IP address of the server on which the request was received - 208.113.183.103

Request_Method

name of the method being used - GET

Request_Protocol

name and version of the protocol with which the request was made - HTTP/1.1

Request_URI

the resource requested on the HTTP request line -- generally the portion of the URL following the scheme and host portion without the query string - /robots.txt

Syntax:

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] ...

Populates HTTP_MY_ Variables with mod_setenvif variable values

SetEnvIfNoCase Remote_Host "(.*)" HTTP_MY_REMOTE_HOST=$1
SetEnvIfNoCase Remote_Addr "(.*)" HTTP_MY_REMOTE_ADDR=$1
SetEnvIfNoCase Server_Addr "(.*)" HTTP_MY_SERVER_ADDR=$1
SetEnvIfNoCase Request_Method "(.*)" HTTP_MY_REQUEST_METHOD=$1
SetEnvIfNoCase Request_Protocol "(.*)" HTTP_MY_REQUEST_PROTOCOL=$1
SetEnvIfNoCase Request_URI "(.*)" HTTP_MY_REQUEST_URI=$1

Set REMOTE_HOST to HTTP_HOST

Sets REMOTE_HOST to www.askapache.com if Remote_Addr=208.113.183.103. This can be useful if your server doesn't automatically do a reverse lookup on a remote address, so this way you can tell if the request was internal/from your server.

SetEnvIf Remote_Addr 208\.113\.183\.103 REMOTE_HOST=www.askapache.com

Allows only if HOST Header is present in request

SetEnvIfNoCase ^HOST$ .+ HTTP_MY_HAS_HOST
Order Deny,Allow
Deny from All
Allow from env=HTTP_MY_HAS_HOST

or

SetEnvIfNoCase Host .+ HTTP_MY_HAS_HOST
Order Deny,Allow
Deny from All
Allow from env=HTTP_MY_HAS_HOST

Add values from HTTP Headers

SetEnvIfNoCase ^If-Modified-Since$ "(.+)" HTTP_IF_MODIFIED_SINCE=$1
SetEnvIfNoCase ^If-None-Match$ "(.+)" HTTP_IF_NONE_MATCH=$1
SetEnvIfNoCase ^Cache-Control$ "(.+)" HTTP_CACHE_CONTROL=$1
SetEnvIfNoCase ^Connection$ "(.+)" HTTP_CONNECTION=$1
SetEnvIfNoCase ^Keep-Alive$ "(.+)" HTTP_KEEP_ALIVE=$1
SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1
SetEnvIfNoCase ^Cookie$ "(.+)" HTTP_MY_COOKIE=$1

Set the REDIRECT_STATUS for Interpreter Security

This is useful in disallowing direct access to interpreters like shell scripts, cgi scripts, and other interpreters. Only works this way if you have a static IP for your server. So the only way to access these files is by instructing the server itself to request the file, using an Action directive or by requesting the file through a .php or other script using curl or wget, or something like fsockopen.

<filesMatch "\.(cgi|sh|pl)$">
SetEnvIfNoCase Remote_Addr 208\.113\.183\.103 REDIRECT_STATUS
 
Order Deny,Allow
Deny from All
Allow from env=REDIRECT_STATUS
</filesMatch>

Block Bad Bots

Can be useful if your site is getting hammered by spambots. Some nice examples from around the net are at Fight Blog Spam With Apache... Keep in mind the HTTP_USER_AGENT is directly from the client, so its easy to spoof / change. Instead use mod_security for a much better solution.

SetEnvIfNoCase User-Agent "^Bandit" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
 
Order Allow,Deny
Allow from All
Deny from env=bad_bot

Allow Search robots

This does the opposite of above, allowing ONLY these web robots access. Other than rogue robots, configuring your robots.txt file correctly will keep most robots where you want them.

SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot
 
Order Deny,Allow
Deny from All
Allow from env=search_robot

SetEnvIf Directive

The SetEnvIf directive defines environment variables based on attributes of the request. The attribute specified in the first argument can be one of three things:

1. An HTTP request header field (see RFC2616 for more information about these); for example: Host, User-Agent, Referer, and Accept-Language. A regular expression may be used to specify a set of request headers.
2. One of the following aspects of the request:
   • Remote_Host - the hostname (if available) of the client making the request
   • Remote_Addr - the IP address of the client making the request
   • Server_Addr - the IP address of the server on which the request was received (only with versions later than 2.0.43)
   • Request_Method - the name of the method being used (GET, POST, et cetera)
   • Request_Protocol - the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.)
   • Request_URI - the resource requested on the HTTP request line -- generally the portion of the URL following the scheme and host portion without the query string. See the RewriteCond directive of mod_rewrite for extra information on how to match your query string.
3. The name of an environment variable in the list of those associated with the request. This allows SetEnvIf directives to test against the result of prior matches. Only those environment variables defined by earlier SetEnvIf[NoCase] directives are available for testing in this manner. 'Earlier' means that they were defined at a broader scope (such as server-wide) or previously in the current directive's scope. Environment variables will be considered only if there was no match among request characteristics and a regular expression was not used for the attribute.

The second argument (regex) is a regular expression. If the regex matches against the attribute, then the remainder of the arguments are evaluated.

The rest of the arguments give the names of variables to set, and optionally values to which they should be set. These take the form of

1. varname
2. !varname
3. varname=value

In the first form, the value will be set to "1". The second will remove the given variable if already defined, and the third will set the variable to the literal value given by value. Since version 2.0.51 Apache will recognize occurrences of $1..$9 within value and replace them by parenthesized subexpressions of regex.

SetEnvIf Example:

SetEnvIf Request_URI "\.gif$" object_is_image=gif
SetEnvIf Request_URI "\.jpg$" object_is_image=jpg
SetEnvIf Request_URI "\.xbm$" object_is_image=xbm
SetEnvIf Referer www\.askapache\.com intra_site_referral
SetEnvIf object_is_image xbm XBIT_PROCESSING=1
SetEnvIf ^SETENVIF*  ^[a-z].*  HAS_SETENVIF

The first three will set the environment variable object_is_image if the request was for an image file, and the fourth sets intra_site_referral if the referring page was somewhere on the www.askapache.com Web site.

The last example will set environment variable HAS_SETENVIF if the request contains any headers that begin with "SETENVIF" whose values begins with any character in the set [a-z].

htaccess Guide Sections

• htaccess tricks for Webmasters
• HTTP Header control with htaccess
• PHP on Apache tips and tricks
• SEO Redirects without mod_rewrite
• mod_rewrite examples, tips, and tricks
• HTTP Caching and Site Speedups
• Authentication on Apache
• htaccess Security Tricks and Tips
• SSL tips and examples
• Variable Fun (mod_env) Section
• .htaccess Security with MOD_SECURITY
• SetEnvIf and SetEnvIfNoCase Examples

« .htaccess Security with MOD_SECURITY | .htaccess Tutorial Index

Htaccess SetEnvIf and SetEnvIfNoCase Examples originally appeared on AskApache.com

This article shows 2 methods of blocking this entire list of bad robots and web scrapers with .htaccess files using SetEnvIfNoCase or using RewriteRules with mod_rewrite

Blocking Bad Robots and Web Scrapers with RewriteRules

ErrorDocument 403 /403.html
 
RewriteEngine On
RewriteBase /
 
# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]
 
# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]
 
# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
 
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

Alternate RewriteCond Rules

RewriteEngine on
 
#Block spambots
RewriteCond %{HTTP:User-Agent} (?:Alexibot|Art-Online|asterias|BackDoorbot|Black.Hole|\
BlackWidow|BlowFish|botALot|BuiltbotTough|Bullseye|BunnySlippers|Cegbfeieh|Cheesebot|\
CherryPicker|ChinaClaw|CopyRightCheck|cosmos|Crescent|Custo|DISCo|DittoSpyder|DownloadsDemon|\
eCatch|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|ExpresssWebPictures|ExtractorPro|\
EyeNetIE|FlashGet|Foobot|FrontPage|GetRight|GetWeb!|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|\
Harvest|hloader|HMView|httplib|HTTrack|humanlinks|ImagesStripper|ImagesSucker|IndysLibrary|\
InfonaviRobot|InterGET|Internet\sNinja|Jennybot|JetCar|JOC\sWeb\sSpider|Kenjin.Spider|Keyword.Density|\
larbin|LeechFTP|Lexibot|libWeb/clsHTTP|LinkextractorPro|LinkScan/8.1a.Unix|LinkWalker|lwp-trivial|\
Mass\sDownloader|Mata.Hari|Microsoft.URL|MIDown\stool|MIIxpc|Mister.PiX|Mister\sPiX|moget|\
Mozilla/3.Mozilla/2.01|Mozilla.*NEWT|Navroad|NearSite|NetAnts|NetMechanic|NetSpider|Net\sVampire|\
NetZIP|NICErsPRO|NPbot|Octopus|Offline.Explorer|Offline\sExplorer|Offline\sNavigator|Openfind|\
Pagerabber|Papa\sFoto|pavuk|pcBrowser|Program\sShareware\s1|ProPowerbot/2.14|ProWebWalker|ProWebWalker|\
psbot/0.1|QueryN.Metasearch|ReGet|RepoMonkey|RMA|SiteSnagger|SlySearch|SmartDownload|Spankbot|spanner|\
Superbot|SuperHTTP|Surfbot|suzuran|Szukacz/1.4|tAkeOut|Teleport|Teleport\sPro|Telesoft|The.Intraformant|\
TheNomad|TightTwatbot|Titan|toCrawl/UrlDispatcher|toCrawl/UrlDispatcher|True_Robot|turingos|\
Turnitinbot/1.5|URLy.Warning|VCI|VoidEYE|WebAuto|WebBandit|WebCopier|WebEMailExtrac.*|WebEnhancer|\
WebFetch|WebGo\sIS|Web.Image.Collector|Web\sImage\sCollector|WebLeacher|WebmasterWorldForumbot|\
WebReaper|WebSauger|Website\seXtractor|Website.Quester|Website\sQuester|Webster.Pro|WebStripper|\
Web\sSucker|WebWhacker|WebZip|Wget|Widow|[Ww]eb[Bb]andit|WWW-Collector-E|WWWOFFLE|\
Xaldon\sWebSpider|Xenu's|Zeus) [NC]
RewriteRule .? - [F]

Block Bad Bots with SetEnvIfNoCase

ErrorDocument 403 /403.html
 
# IF THE UA STARTS WITH THESE
SetEnvIfNoCase ^User-Agent$ .*(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(libwww-perl|aesop_com_spiderman) HTTP_SAFE_BADBOT
Deny from env=HTTP_SAFE_BADBOT

Original Bad Bot / Web Scraper List

1. WebBandit
2. 2icommerce
3. Accoona
4. ActiveTouristBot
5. adressendeutschland
6. aipbot
7. Alexibot
8. Alligator
9. AllSubmitter
10. almaden
11. anarchie
12. Anonymous
13. Apexoo
14. Aqua_Products
15. asterias
16. ASSORT
17. ATHENS
18. AtHome
19. Atomz
20. attache
21. autoemailspider
22. autohttp
23. b2w
24. bew
25. BackDoorBot
26. Badass
27. Baiduspider
28. Baiduspider+
29. BecomeBot
30. berts
31. Bitacle
32. Biz360
33. Black.Hole
34. BlackWidow
35. bladder fusion
36. Blog Checker
37. BlogPeople
38. Blogshares Spiders
39. Bloodhound
40. BlowFish
41. Board Bot
42. Bookmark search tool
43. BotALot
44. BotRightHere
45. Bot mailto:craftbot@yahoo.com
46. Bropwers
47. Browsezilla
48. BuiltBotTough
49. Bullseye
50. BunnySlippers
51. Cegbfeieh
52. CFNetwork
53. CheeseBot
54. CherryPicker
55. Crescent
56. charlotte/
57. ChinaClaw
58. Convera
59. Copernic
60. CopyRightCheck
61. cosmos
62. Crescent
63. c-spider
64. curl
65. Custo
66. Cyberz
67. DataCha0s
68. Daum
69. Deweb
70. Digger
71. Digimarc
72. digout4uagent
73. DIIbot
74. DISCo
75. DittoSpyder
76. DnloadMage
77. Download
78. dragonfly
79. DreamPassport
80. DSurf
81. DTS Agent
82. dumbot
83. DynaWeb
84. e-collector
85. EasyDL
86. EBrowse
87. eCatch
88. ecollector
89. edgeio
90. efp@gmx.net
91. EirGrabber
92. Email Extractor
93. EmailCollector
94. EmailSiphon
95. EmailWolf
96. EmeraldShield
97. Enterprise_Search
98. EroCrawler
99. ESurf
100. Eval
101. Everest-Vulcan
102. Exabot
103. Express
104. Extractor
105. ExtractorPro
106. EyeNetIE
107. FairAd
108. fastlwspider
109. fetch
110. FEZhead
111. FileHound
112. findlinks
113. Flaming AttackBot
114. FlashGet
115. FlickBot
116. Foobot
117. Forex
118. Franklin Locator
119. FreshDownload
120. FrontPage
121. FSurf
122. Gaisbot
123. Gamespy_Arcade
124. genieBot
125. GetBot
126. Getleft
127. GetRight
128. GetWeb!
129. Go!Zilla
130. Go-Ahead-Got-It
131. GOFORITBOT
132. GrabNet
133. Grafula
134. grub
135. Harvest
136. Hatena Antenna
137. heritrix
138. HLoader
139. HMView
140. holmes
141. HooWWWer
142. HouxouCrawler
143. HTTPGet
144. httplib
145. HTTPRetriever
146. HTTrack
147. humanlinks
148. IBM_Planetwide
149. iCCrawler
150. ichiro
151. iGetter
152. Image Stripper
153. Image Sucker
154. imagefetch
155. imds_monitor
156. IncyWincy
157. Industry Program
158. Indy
159. InetURL
160. InfoNaviRobot
161. InstallShield DigitalWizard
162. InterGET
163. IRLbot
164. Iron33
165. ISSpider
166. IUPUI Research Bot
167. Jakarta
168. java/
169. JBH Agent
170. JennyBot
171. JetCar
172. jeteye
173. jeteyebot
174. JoBo
175. JOC Web Spider
176. Kapere
177. Kenjin
178. Keyword Density
179. KRetrieve
180. ksoap
181. KWebGet
182. LapozzBot
183. larbin
184. leech
185. LeechFTP
186. LeechGet
187. leipzig.de
188. LexiBot
189. libWeb
190. libwww-FM
191. libwww-perl
192. LightningDownload
193. LinkextractorPro
194. Linkie
195. LinkScan
196. linktiger
197. LinkWalker
198. lmcrawler
199. LNSpiderguy
200. LocalcomBot
201. looksmart
202. LWP
203. Mac Finder
204. Mail Sweeper
205. mark.blonin
206. MaSagool
207. Mass
208. Mata Hari
209. MCspider
210. MetaProducts Download Express
211. Microsoft Data Access
212. Microsoft URL Control
213. MIDown
214. MIIxpc
215. Mirror
216. Missauga
217. Missouri College Browse
218. Mister
219. Monster
220. mkdb
221. moget
222. Moreoverbot
223. mothra/netscan
224. MovableType
225. Mozi!
226. Mozilla/22
227. Mozilla/3.0 (compatible)
228. Mozilla/5.0 (compatible; MSIE 5.0)
229. MSIE_6.0
230. MSIECrawler
231. MSProxy
232. MVAClient
233. MyFamilyBot
234. MyGetRight
235. nameprotect
236. NASA Search
237. Naver
238. Navroad
239. NearSite
240. NetAnts
241. netattache
242. NetCarta
243. NetMechanic
244. NetResearchServer
245. NetSpider
246. NetZIP
247. Net Vampire
248. NEWT ActiveX
249. Nextopia
250. NICErsPRO
251. ninja
252. NimbleCrawler
253. noxtrumbot
254. NPBot
255. Octopus
256. Offline
257. OK Mozilla
258. OmniExplorer
259. OpaL
260. Openbot
261. Openfind
262. OpenTextSiteCrawler
263. Oracle Ultra Search
264. OutfoxBot
265. P3P
266. PackRat
267. PageGrabber
268. PagmIEDownload
269. panscient
270. Papa Foto
271. pavuk
272. pcBrowser
273. perl
274. PerMan
275. PersonaPilot
276. PHP version
277. PlantyNet_WebRobot
278. playstarmusic
279. Plucker
280. Port Huron
281. Program Shareware
282. Progressive Download
283. ProPowerBot
284. prospector
285. ProWebWalker
286. Prozilla
287. psbot
288. psycheclone
289. puf
290. PushSite
291. PussyCat
292. PuxaRapido
293. Python-urllib
294. QuepasaCreep
295. QueryN
296. Radiation
297. RealDownload
298. RedCarpet
299. RedKernel
300. ReGet
301. relevantnoise
302. RepoMonkey
303. RMA
304. Rover
305. Rsync
306. RTG30
307. Rufus
308. SAPO
309. SBIder
310. scooter
311. ScoutAbout
312. script
313. searchpreview
314. searchterms
315. Seekbot
316. Serious
317. Shai
318. shelob
319. Shim-Crawler
320. SickleBot
321. sitecheck
322. SiteSnagger
323. Slurpy Verifier
324. SlySearch
325. SmartDownload
326. sna-
327. snagger
328. Snoopy
329. sogou
330. sootle
331. So-net” bat_bot
332. SpankBot” bat_bot
333. spanner” bat_bot
334. SpeedDownload
335. Spegla
336. Sphere
337. Sphider
338. SpiderBot
339. sproose
340. SQ Webscanner
341. Sqworm
342. Stamina
343. Stanford
344. studybot
345. SuperBot
346. SuperHTTP
347. Surfbot
348. SurfWalker
349. suzuran
350. Szukacz
351. tAkeOut
352. TALWinHttpClient
353. tarspider
354. Teleport
355. Telesoft
356. Templeton
357. TestBED
358. The Intraformant
359. TheNomad
360. TightTwatBot
361. Titan
362. toCrawl/UrlDispatcher
363. True_Robot
364. turingos
365. TurnitinBot
366. Twisted PageGetter
367. UCmore
368. UdmSearch
369. UMBC
370. UniversalFeedParser
371. URL Control
372. URLGetFile
373. URLy Warning
374. URL_Spider_Pro
375. UtilMind
376. vayala
377. vobsub
378. VCI
379. VoidEYE
380. VoilaBot
381. voyager
382. w3mir
383. Web Image Collector
384. Web Sucker
385. Web2WAP
386. WebaltBot
387. WebAuto
388. WebBandit
389. WebCapture
390. webcollage
391. WebCopier
392. WebCopy
393. WebEMailExtrac
394. WebEnhancer
395. WebFetch
396. WebFilter
397. WebFountain
398. WebGo
399. WebLeacher
400. WebMiner
401. WebMirror
402. WebReaper
403. WebSauger
404. WebSnake
405. Website
406. WebStripper
407. WebVac
408. webwalk
409. WebWhacker
410. WebZIP
411. Wells Search
412. WEP Search 00
413. WeRelateBot
414. Wget
415. WhosTalking
416. Widow
417. Wildsoft Surfer
418. WinHttpRequest
419. WinHTTrack
420. WUMPUS
421. WWWOFFLE
422. wwwster
423. WWW-Collector
424. Xaldon
425. Xenu's
426. Xenus
427. XGET
428. Y!TunnelPro
429. YahooYSMcm
430. YaDirectBot
431. Yeti
432. Zade
433. ZBot
434. zerxbot
435. Zeus
436. ZyBorg

Blocking Bad Bots and Scrapers with .htaccess originally appeared on AskApache.com

Implementing an effective SEO robots.txt file for WordPress will help your blog to rank higher in Search Engines, receive higher paying relevant Ads, and increase your blog traffic. Using a robots.txt file gives you a search engine robots point of view... Sweet! Looking for the most updated robots.txt? Just look at mine, I don't slack.

WordPress robots.txt SEO

Here are some robots.txt files used with WordPress on this blog. For instance, I am disallowing /category/ in the robots.txt file below because askapache.com/category/htaccess/ is the same as askapache.com/htaccess/, and that would be duplicate content. Adding a 301 Redirect using mod_rewrite or RedirectMatch can further protect myself from this duplicate content issue.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /tag
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/
 
User-agent: Mediapartners-Google
Allow: /
 
User-agent: Adsbot-Google
Allow: /
 
User-agent: Googlebot-Image
Allow: /
 
User-agent: Googlebot-Mobile
Allow: /
 
User-agent: ia_archiver-web.archive.org
Disallow: /
 
Sitemap: http://www.askapache.com/sitemap.xml
 
#               __                          __
#   ____ ______/ /______ _____  ____ ______/ /_  ___
#  / __ `/ ___/ //_/ __ `/ __ \/ __ `/ ___/ __ \/ _ \
# / /_/ (__  ) ,< / /_/ / /_/ / /_/ / /__/ / / /  __/
# \__,_/____/_/|_|\__,_/ .___/\__,_/\___/_/ /_/\___/
#                   /_/
#

static.askapache.com/robots.txt

User-agent: *
Disallow:
Allow: /*
 
User-agent: ia_archiver
Disallow: /
 
User-agent: duggmirror
Disallow: /

Google Recommendations

Use robots.txt - Webmaster Guidelines

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.

Troubleshooting tips part IIb: Ad relevance and targeting continued

To follow up on our previous post about ad relevance and targeting, let's look at some other reasons why you may experience ad targeting issues on your site.

Have you blocked the AdSense crawler's access to your pages?

The AdSense crawler is an automated program that scans your web pages and tracks content for indexing. Sometimes we don't crawl pages because the AdSense crawler doesn't have access to your pages, in which case we're unable to determine their content and show relevant ads. Here are a few specific instances when our crawler can't access a site: If you use a robots.txt file which regulates the crawler access to your page. In this case, you can grant the AdSense crawler access by adding these lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

Eliminate Duplicate Content

Duplicate content generally refers to substantive blocks of content within or across domains that either completely match other content or are appreciably similar. Mostly, this is not deceptive in origin. Examples of non-malicious duplicate content could include:

• Discussion forums that can generate both regular and stripped-down pages targeted at mobile devices
• Store items shown or linked via multiple distinct URLs
• Printer-only versions of web pages

However, in some cases, content is deliberately duplicated across domains in an attempt to manipulate search engine rankings or win more traffic. Deceptive practices like this can result in a poor user experience, when a visitor sees substantially the same content repeated within a set of search results.

Google tries hard to index and show pages with distinct information. This filtering means, for instance, that if your site has a "regular" and "printer" version of each article, and neither of these is blocked in robots.txt or with a noindex meta tag, we'll choose one of them to list. In the rare cases in which Google perceives that duplicate content may be shown with intent to manipulate our rankings and deceive our users, we'll also make appropriate adjustments in the indexing and ranking of the sites involved. As a result, the ranking of the site may suffer, or the site might be removed entirely from the Google index, in which case it will no longer appear in search results.

Prevent page from being indexed

Pages you block in this way may still be added to the Google index if other sites link to them. As a result, the URL of the page and, potentially, other publicly available information can appear in Google search results. However, no content from your pages will be crawled, indexed, or displayed.

To entirely prevent a page from being added to the Google index even if other sites link to it, use a noindex meta tag, and ensure that the page does not appear in robots.txt. When Googlebot crawls the page, it will recognize the noindex meta tag and drop the URL from the index.

Prevent content being indexed or remove content from Google's index?

You can instruct us not to include content from your site in our index or to remove content from your site that is currently in our index in the following ways:

• Remove your entire website or part of your website using a robots.txt file.
• Remove individual pages of your website using a robots meta tag.
• Remove cached copies of your pages using a robots meta tag.
• Remove snippets that appear below your page's title in our search results and describe the content of your page.
• Remove outdated pages by returning the proper server response.
• Remove images from Google Image Search using a robots.txt file.
• Remove blog entries from Google Blog Search.
• Remove a feed from our user-agent Feedfetcher, which provides content to our feed readers.
• Remove transcoded versions of your pages (pages we've reformatted for mobile browsers).

Google User-agents

Adsbot-Google

crawls pages to measure AdWords landing page quality

Googlebot

crawl pages from googles web and news index

Googlebot-Image

crawls pages for the image index

Googlebot-Mobile

crawls pages for the mobile index

Mediapartners-Google

crawls pages to determine AdSense content

Good Robots.txt Articles

1. How Google Crawls My Site
2. Using the robots.txt analysis tool
3. Controlling how search engines access and index your website
4. Controlling Access with robots.txt
5. Removing duplicate search engine content using robots.txt - Mark Wilson
6. Revisiting robots.txt - Twenty Steps

Robots Meta Tags

Using the robots meta tag

Robots Meta Examples

Stop all robots from indexing a page on your site, but still follow the links on the page

<meta name="robots" content="noindex,follow" />

Allow other robots to index the page on your site, preventing only Googles bots from indexing the page

<meta name="googlebot" content="noindex,follow" />

Allow robots to index the page on your site but not to follow outgoing links

<meta name="robots" content="nofollow" />

header.php Trick for Conditional Robots Meta

Add this to your header.php

<?php if(is_single() || is_page() || is_category() || is_home()) { ?>
  <meta name="robots" content="all,noodp" />
<?php } ?>
<?php if(is_archive()) { ?>
  <meta name="robots" content="noarchive,noodp" />
<?php } ?>
<?php if(is_search() || is_404()) { ?>
  <meta name="robots" content="noindex,noarchive" />
<?php } ?>

---

Robots.txt footnote Alexa, Compete, and Quantcast are all guilty of firewalling unknown friendly search engine agents at the front gate. These sites that monitor the Internet should be the most in the know that unfriendly agents cloak as humans and will come in no matter what. So the general rule of thumb is that robots.txt directives are only for the good agents anyway.

---

Robots.txt References

1. Robots.txt optimization
2. The Web Robots Pages
3. W3.org - Notes on helping search engines index your Web site
4. Wikipedia robots.txt page
5. Inside Google Sitemaps: Using a robots.txt file

wordpress robots.txt

Updated robots.txt for WordPress originally appeared on AskApache.com

]]> http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html/feed 52 http://www.askapache.com/shellscript/php5-custom-install.html http://www.askapache.com/shellscript/php5-custom-install.html#comments Mon, 17 Dec 2007 04:11:31 +0000 AskApache http://www.askapache.com/dreamhost/php5-custom-install.html Today I successfully learned how to compile and run multiple custom php installations for a DreamHost account, and to get it working I came upon a simple shell script that I made a couple changes to.

]]> AskApache.com

Today I successfully learned how to compile and run multiple custom php installations for a DreamHost account, and to get it working I came upon a simple shell script that I made a couple changes to.

This script downloads the correct files, php extensions, and source code that you will need to get a custom php5 environment set up on DH. Its a rather dangerous file though if you don't know what you are doing, because compiling your own custom php-cgi for your web sites isn't exactly a common thing. This file will delete a couple of your directories if you aren't careful, I just posted it here because I enjoyed hacking this out and this is a pretty cool example of some of the things you can do with the unix shell.

Its fun to rotate between languages like that, although shell programming isn't exactly the most exciting thing compared to python, ruby, and other great Object->Oriented languages taking over. But they do all run in the shell :)

These are alternative PHP 5 install scripts created by Charles Wiltgen, who created them to work around problems he was having with the "main" PHP 5 install script.

Discussed in more detail on the DreamHost Customer Wiki.

Functions in Shell Scripts

Two changes I made were adding some functionality to the code with these two tiny gunctions, one downloads compressed files, the other decompresses them.

function aa_unpack () {
  # compressed, tar and gzip files to DISTDIR
  if [ -f $DISTDIR/$1* ] ; then
    echo Extracting "$1";
    zcat ${DISTDIR}/$1* | tar -xvf - &>/dev/null;
    echo Done.;  echo; wait
  fi
}
 
function askapache_grab () {
  #saves file to SRCDIR
    echo `basename $1`
  curl -L --retry 20 --max-time 1800 --retry-delay 30 -# -f --max-redirs 4 --remote-name "$1"
}

php5-install-prep.sh - Source Code

#!/bin/sh
 
# Version 0.6, 2007-12-16
#
# - Updated 2007-12-16 by AskApache (www.askapache.com)
#   - Implemented functions to fetch the URI and decompress it
#   - Added a couple more error-checks
#   - Replaced wget with cURL
#   - Added more to help keep it from getting killed
#  - Updated to php-5.2.3, curl-7.17.1, freetype-2.3.5
# - Updated 2007-01-15 by Charles Wiltgen (charles@wiltgen.net)
#   - Make "nicer" to help keep it from getting killed by DreamHost
#   - Make less verbose to keep signal-to-noise level high
# - Updated 2006-12-25 by Carl McDade (hiveminds.co.uk)
#   - Allow memory limit and freetype
 
# Abort on any errors
set -e
 
# The domain in which to install the PHP CGI script.
export DOMAIN="askapache.com"
 
# Where do you want all this stuff built? I'd recommend picking a local filesystem.
# ***Don't pick a directory that already exists!***
SRCDIR=${HOME}/source
 
# And where should it be installed?
INSTALLDIR=${HOME}/php5
 
# Set DISTDIR to somewhere persistent, if you plan to muck around with this
# script and run it several times!
DISTDIR=${HOME}/dist
 
# Update version information here.
PHP5="php-5.2.3"
LIBICONV="libiconv-1.11"
LIBMCRYPT="libmcrypt-2.5.7"
LIBXML2="libxml2-2.6.27"
LIBXSLT="libxslt-1.1.18"
MHASH="mhash-0.9.7.1"
ZLIB="zlib-1.2.3"
CURL="curl-7.17.1"
LIBIDN="libidn-0.6.8"
CCLIENT="imap-2004g"
CCLIENT_DIR="imap-2004g"
FREETYPE="freetype-2.3.5"
 
# Push the install dir's bin directory into the path
export PATH=${INSTALLDIR}/bin:$PATH
 
function aa_unpack () {
  # compressed, tar and gzip files to DISTDIR
  if [ -f $DISTDIR/$1* ] ; then
    echo Extracting "$1";
    zcat ${DISTDIR}/$1* | tar -xvf - &>/dev/null;
    echo Done.;  echo; wait
  fi
}
 
function askapache_grab () {
  #saves file to SRCDIR
    echo `basename $1`
  curl -L --retry 20 --max-time 1800 --retry-delay 30 -# -f --max-redirs 4 --remote-name "$1"
}
 
echo
echo --------------------------------------------------
echo --   Run this script before php5-install.sh     --
echo --------------------------------------------------
echo
echo - Downloads and unpacks all prerequisite packages
echo - **SRCDIR and DISTDIR will be deleted**
echo
read -p  "        (Press any key to continue)" temp;
echo;echo
 
# cleanup to remove source and dist directories if present
if [ -d "$SRCDIR" ] || [ -d "$DISTDIR" ];then
  echo
  echo --- Cleaning up any previous attempts ---
  rm -rf $SRCDIR $DISTDIR &>/dev/null
  echo Done.
  echo
  wait
fi
 
#setup directories
mkdir -p ${SRCDIR} ${INSTALLDIR} ${DISTDIR} &>/dev/null
 
# Get all the required packages
echo;echo
echo --- Downloading all required packages ---
echo
 
cd ${DISTDIR}
askapache_grab http://us.php.net/distributions/${PHP5}.tar.gz
askapache_grab http://mirrors.usc.edu/pub/gnu/libiconv/${LIBICONV}.tar.gz
askapache_grab http://umn.dl.sourceforge.net/sourceforge/mcrypt/${LIBMCRYPT}.tar.gz
askapache_grab ftp://xmlsoft.org/libxml2/${LIBXML2}.tar.gz
askapache_grab ftp://xmlsoft.org/libxml2/${LIBXSLT}.tar.gz
askapache_grab http://umn.dl.sourceforge.net/sourceforge/mhash/${MHASH}.tar.gz
askapache_grab http://www.zlib.net/${ZLIB}.tar.gz
askapache_grab http://curl.askapache.com/download/${CURL}.tar.gz
askapache_grab http://easynews.dl.sourceforge.net/sourceforge/freetype/${FREETYPE}.tar.gz
askapache_grab ftp://alpha.gnu.org/pub/gnu/libidn/${LIBIDN}.tar.gz
askapache_grab ftp://ftp.cac.washington.edu/imap/old/${CCLIENT}.tar.Z
wait
echo Done.
 
# Extract the files from the required packages.
echo;echo;echo
echo --- Unpacking downloaded archives. This process may take several minutes! ---
echo
 
cd ${SRCDIR}
aa_unpack ${PHP5}
aa_unpack ${LIBICONV}
aa_unpack ${LIBMCRYPT}
aa_unpack ${LIBXML2}
aa_unpack ${LIBXSLT}
aa_unpack ${MHASH}
aa_unpack ${ZLIB}
aa_unpack ${CURL}
aa_unpack ${LIBIDN}
aa_unpack ${CCLIENT}
aa_unpack ${FREETYPE}
wait
 
echo --------------------------------------------------
echo -- Done downloading and unpacking prerequisites --
echo --------------------------------------------------
 
exit 0;

php5-install-prep.sh - Output

[root@~/.^*][~/dm/e]
$ ./php5-install-prep.sh
 
--------------------------------------------------
-- Run this script before php5-install.sh --
--------------------------------------------------
 
- Downloads and unpacks all prerequisite packages
- **SRCDIR and DISTDIR will be deleted**
 
        (Press any key to continue)
 
--- Cleaning up any previous attempts ---
Done.
 
--- Downloading all required packages ---
 
php-5.2.3.tar.gz
######################################################################## 100.0%
libiconv-1.11.tar.gz
######################################################################## 100.0%
libmcrypt-2.5.7.tar.gz
######################################################################## 100.0%
libxml2-2.6.27.tar.gz
######################################################################## 100.0%
libxslt-1.1.18.tar.gz
######################################################################## 100.0%
mhash-0.9.7.1.tar.gz
######################################################################## 100.0%
zlib-1.2.3.tar.gz
######################################################################## 100.0%
curl-7.17.1.tar.gz
######################################################################## 100.0%
freetype-2.3.5.tar.gz
######################################################################## 100.0%
libidn-0.6.8.tar.gz
######################################################################## 100.0%
imap-2004g.tar.Z
######################################################################## 100.0%
Done.
 
--- Unpacking downloaded archives. This process may take several minutes! ---
 
Extracting php-5.2.3
Done.
 
Extracting libiconv-1.11
Done.
 
Extracting libmcrypt-2.5.7
Done.
 
Extracting libxml2-2.6.27
Done.
 
Extracting libxslt-1.1.18
Done.
 
Extracting mhash-0.9.7.1
Done.
 
Extracting zlib-1.2.3
Done.
 
Extracting curl-7.17.1
Done.
 
Extracting libidn-0.6.8
Done.
 
Extracting imap-2004g
Done.
 
Extracting freetype-2.3.5
Done.
 
--------------------------------------------------
-- Done downloading and unpacking prerequisites --
--------------------------------------------------

PHP5 Custom Install Shell Script Example originally appeared on AskApache.com

]]> http://www.askapache.com/shellscript/php5-custom-install.html/feed 1 http://www.askapache.com/seo/seo-with-robotstxt.html http://www.askapache.com/seo/seo-with-robotstxt.html#comments Sat, 20 Oct 2007 16:59:38 +0000 AskApache http://www.askapache.com.com/seo/seo-with-robotstxt.html AskApache.com

robots.txt Search Engine Optimization is simply using robots.txt for your blog, wordpress, or phpbb. WordPress Optimized robots.txt and meta tags

---

See the Updated WordPress robots.txt file

• Google Robots.txt Info and Recommendations
  • Googlebot and Robots.txt SEO Info
  • Google User-agents
  • Removing Old/wrong content from google
  • Google Robots.txt Articles
• Example Robots.txt files
  • robots.txt for WordPress
  • robots.txt for phpBB

Google Robots.txt Info and Recommendations

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.

Googlebot and Robots.txt SEO Info

When deciding which pages to crawl, Googlebot goes in this order

1. Googlebot will obey the first record in the robots.txt file with a User-agent starting with "Googlebot."
2. If no "Googlebot User-agent exists, it will obey the first entry with a User-agent of "*"

Google User-agents

Googlebot

crawl pages from our web index and our news index

Googlebot-Mobile

crawls pages for our mobile index

Googlebot-Image

crawls pages for our image index

Mediapartners-Google

crawls pages to determine AdSense content. We only use this bot to crawl your site if you show AdSense ads on your site.

Adsbot-Google

crawls pages to measure AdWords landing page quality. We only use this bot if you use Google AdWords to advertise your site. Find out more about this bot and how to block it from portions of your site.

Removing Old/wrong content from google

1. Create the new page
2. In .htaccess (if Linux) add a RedirectPermanent command
3. DO NOT DELETE THE OLD FILE
4. Update all the links on your website to point to the new page (change the link text while you're at it)
5. Verify that no pages point to the old file (including your sitemap.xml)
6. Add a noindex,nofollow to the old file AND Disallow in your robots.txt
7. Submit your updated sitemap.xml to Google & Yahoo
8. Wait a few weeks
9. When the new page appears in Google, it's safe to delete the old one

Google Sponsored Robots.txt Articles

1. Controlling how search engines access and index your website
2. The Robots Exclusion Protocol
3. robots.txt analysis tool
4. Googlebot
5. Inside Google Sitemaps: Using a robots.txt file
6. All About Googlebot

robots.txt examples

robots.txt for WordPress 2.+

User-agent:  *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: /dh_
Disallow: /about/
Disallow: /contact/
Disallow: /tag/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /contact
Disallow: /manual
Disallow: /manual/*
Disallow: /phpmanual/
Disallow: /category/
 
User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
 
# disallow all files with ? in url
Disallow: /*?*
 
# disable duggmirror
User-agent: duggmirror
Disallow: /
 
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
 
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

robots.txt for phpBB

User-agent: *
Disallow: /cgi-bin/
Disallow: /phpbb/admin/
Disallow: /phpbb/cache/
Disallow: /phpbb/db/
Disallow: /phpbb/images/
Disallow: /phpbb/includes/
Disallow: /phpbb/language/
Disallow: /phpbb/templates/
Disallow: /phpbb/faq.php
Disallow: /phpbb/groupcp.php
Disallow: /phpbb/login.php
Disallow: /phpbb/memberlist.php
Disallow: /phpbb/modcp.php
Disallow: /phpbb/posting.php
Disallow: /phpbb/privmsg.php
Disallow: /phpbb/profile.php
Disallow: /phpbb/search.php
Disallow: /phpbb/viewonline.php
 
User-agent: Googlebot
# disallow files ending with these extensions
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
 
# disallow all files with? in url
Disallow: *mark=*
Disallow: *view=*
 
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
 
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

User-agent: *
Disallow: /stats
Disallow: /dh_
Disallow: /V
Disallow: /z/j/
Disallow: /z/c/
Disallow: /cgi-bin/
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /post
Disallow: /member
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php
Disallow: /templates/
Disallow: /mx_
Disallow: /db/
Disallow: /admin/
Disallow: /cache/
Disallow: /images/
Disallow: /includes/
Disallow: /common.php
Disallow: /index.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /images/
Disallow: /rss.php
 
User-agent: Googlebot
# disallow all files ending with these extensions
Allow: /sitemap.php
Disallow: /*.php$
Allow: /sitemap.php
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
 
# disallow all files with? in url
Disallow: /*?*
Disallow: /*?
 
# disallow all files in /wp- directorys
Disallow: /wp-*/
 
# disallow archiving site
User-agent: ia_archiver
Disallow: /
 
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*.gif$
Allow: /*.png$
Allow: /*.jpeg$
Allow: /*.jpg$
Allow: /*.ico$
Allow: /*.jpg$
Allow: /images
Allow: /z/i/
 
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Allow: /*

Pattern Matching with Google

Matching a sequence of characters using *

You can use an asterisk * to match a sequence of characters.

Block access to all subdirectories that begin with private:

User-Agent: Googlebot
Disallow: /private*/

Block access to all URLs that include a ?

User-agent: *
Disallow: /*?*

Matching the end characters of the URL using $

You can use the $ character to specify matching the end of the URL.

Block any URLs that end with .php

User-Agent: Googlebot
Disallow: /*.php$

You can use this pattern matching in combination with the Allow directive.

Exclude all URLs that contain ? to ensure Googlebot doesn't crawl duplicate pages. URLs that end with a ? DO get crawled

User-agent: *
Allow: /*?$
Disallow: /*?

Disallow:/*? blocks any URL that begins with HOST, followed by any string, followed by a ?, followed by any string

Allow: /*?$ allows any URL that begins with HOST, followed by any string, followed by a ?, with no characters after the ?

SRC

User-Agent Discussion

Blocking a specific User-Agent

Note: Blocking Googlebot blocks all bots that begin with "Googlebot"

Block Googlebot entirely

User-agent: Googlebot
Disallow: /

Allowing a specific User-Agent

Note: Googlebot follows the line directed at it, rather than the line directed at everyone.

Block access to all bots other than "Googlebot"

User-agent: *
Disallow: /
 
User-agent: Googlebot
Disallow:

Googlebot recognizes an extension to the robots.txt standard called Allow, which is opposite of Disallow.

Block all pages inside a subdirectory except for single file

User-Agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

Block Googlebot but allow other Bot

User-agent: Googlebot
Disallow: /
 
User-agent: Googlebot-Mobile
Allow:

SRC

Removing Content From Google

It is better to use <meta name="Googlebot" content="Follow,NoIndex"> on pages that have been indexed if you wish google to drop them. This way it is much faster than blocked using robots.txt.

Note: removing snippets also removes cached pages.

A snippet is a text excerpt that appears below a page's title in our search results and describes the content of the page.

Prevent Google from displaying snippets for your page

<meta NAME="GOOGLEBOT" CONTENT="NOSNIPPET">

Remove an outdated "dead" link

Google updates its entire index automatically on a regular basis. When we crawl the web, we find new pages, discard dead links, and update links automatically. Links that are outdated now will most likely "fade out" of our index during our next crawl.

Note: Please ensure that you return a true 404 error even if you choose to display a more user-friendly body of the HTML page for your visitors. It won't help to return a page that says "File Not Found" if the http headers still return a status code of 200, or normal.

Remove cached pages

Google automatically takes a "snapshot" of each page it crawls and archives it. This "cached" version allows a webpage to be retrieved for your end users if the original page is ever unavailable (due to temporary failure of the page's web server). The cached page appears to users exactly as it looked when Google last crawled it, and we display a message at the top of the page to indicate that it's a cached version. Users can access the cached version by choosing the "Cached" link on the search results page.

Prevent all search engines from showing a "Cached" link for your site

<meta name="robots" content="noarchive" />

Allow other search engines to show a "Cached" link, preventing only Google

<meta name="googlebot" content="noarchive" />

Note: this tag only removes the "Cached" link for the page. Google will continue to index the page and display a snippet.

Remove your entire website

If you wish to exclude your entire website from Google's index

Remove site from search engines and prevent all robots from crawling it in the future

User-agent: *
Disallow: /

Note: Please note that Googlebot does not interpret a 401/403 response ("Unauthorized"/"Forbidden") to a robots.txt fetch as a request not to crawl any pages on the site.

To remove your site from Google only and prevent just Googlebot from crawling your site in the future

User-agent: Googlebot
Disallow: /

Allow Googlebot to index all http pages but no https pages

Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols.

For your http protocol (http://yourserver.com/robots.txt)

User-agent: *
Allow: /

For the https protocol (https://yourserver.com/robots.txt)

User-agent: *
Disallow: /

Remove part of your website

Option 1: Robots.txt

Remove all pages under a particular directory (for example, lems)

User-agent: Googlebot
Disallow: /lems

Remove all files of a specific file type (for example, .gif)

User-agent: Googlebot
Disallow: /*.gif$

To remove dynamically generated pages, you'd use this robots.txt entry

User-agent: Googlebot
Disallow: /*?

Option 2: Meta tags

Another standard, which can be more convenient for page-by-page use, involves adding a META tag to an HTML page to tell robots not to index the page. This standard is described at http://www.robotstxt.org/wc/exclusion.html#meta.

Prevent all robots from indexing a page on your site

<meta name="robots" content="noindex, nofollow" />

Allow other robots to index the page on your site, preventing only Google's robots from indexing the page

<meta name="googlebot" content="noindex, nofollow" />

Allow robots to index the page on your site but instruct them not to follow outgoing links

<meta name="robots" content="nofollow" />

Remove an image from Google's Image Search

Want Google to exclude the dogs.jpg image that appears on your site at www.yoursite.com/images/dogs.jpg

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

Remove all the images on your site from our index

User-agent: Googlebot-Image
Disallow: /

Remove all files of a specific file type (for example, to include .jpg but not .gif images)

User-agent: Googlebot-Image
Disallow: /*.gif$

Note: If you believe your request is urgent and cannot wait until the next time Google crawls your site, use our automatic URL removal system. In order for this automated process to work, the webmaster or SEO agency must first create and place a robots.txt file on the site in question.

Google will continue to exclude your site or directories from successive crawls if the robots.txt file exists in the web server root. If you do not have access to the root level of your server, you may place a robots.txt file at the same level as the files you want to remove. Doing this and submitting via the automatic URL removal system will cause a temporary, 180 day removal of the directories specified in your robots.txt file from the Google index, regardless of whether you remove the robots.txt file after processing your request. (Keeping the robots.txt file at the same level would require you to return to the URL removal system every 180 days to reissue the removal.)

Remove a blog from Blog Search

Only blogs with site feeds will be included in Blog Search. If you'd like to prevent your feed from being crawled, make use of a robots.txt file or meta tags (NOINDEX or NOFOLLOW), as described above. Please note that if you have a feed that was previously included, the old posts will remain in the index even though new ones will not be added.

Remove a RSS or Atom feed

When users add your feed to their Google homepage or Google Reader, Google's Feedfetcher attempts to obtain the content of the feed in order to display it. Since Feedfetcher requests come from explicit action by human users, Feedfetcher has been designed to ignore robots.txt guidelines.

It's not possible for Google to restrict access to a publicly available feed. If your feed is provided by a blog hosting service, you should work with them to restrict access to your feed. Check those sites' help content for more information (e.g., Blogger, LiveJournal, or Typepad).

Remove transcoded pages

Google Web Search on mobile phones allows users to search all the content in the Google index for desktop web browsers. Because this content isn't written specifically for mobile phones and devices and thus might not display properly, Google automatically translates (or "transcodes") these pages by analyzing the original HTML code and converting it to a mobile-ready format. To ensure that the highest quality and most useable web page is displayed on your mobile phone or device, Google may resize, adjust, or convert images, text formatting and/or certain aspects of web page functionality.

SRC

To save bandwidth, Googlebot only downloads the robots.txt file once a day or whenever we've fetched many pages from the server. So, it may take a while for Googlebot to learn of changes to your robots.txt file. Also, Googlebot is distributed on several machines. Each of these keeps its own record of your robots.txt file.

Also, there's a small difference between the way Googlebot handles the robots.txt file and the way the robots.txt standard says we should (keeping in mind the distinction between "should" and "must"). The standard says we should obey the first applicable rule, whereas Googlebot obeys the longest (that is, the most specific) applicable rule. This more intuitive practice matches what people actually do, and what they expect us to do.

For example, consider the following robots.txt file:

User-Agent: *
Allow: /
Disallow: /cgi-bin

It's obvious that the webmaster's intent here is to allow robots to crawl everything except the /cgi-bin directory. Consequently, that's what we do.

Tell googlebot not to count certain external links in your ranking

Meta tags can exclude all outgoing links on a page, but you can also instruct Googlebot not to crawl individual links by adding rel="nofollow" to a hyperlink. When Google sees the attribute rel="nofollow" on hyperlinks, those links won't get any credit when we rank websites in our search results. For example a link, This is a great link! could be replaced with I can't vouch for this link.

Other Links

1. Database of Web Robots, Overview

---

# robots.txt, www.nytimes.com 6/29/2006
User-agent: *
Disallow: /pages/college/
Disallow: /college/
Disallow: /library/
Disallow: /learning/
Disallow: /aponline/
Disallow: /reuters/
Disallow: /cnet/
Disallow: /partners/
Disallow: /archives/
Disallow: /indexes/
Disallow: /thestreet/
Disallow: /nytimes-partners/
Disallow: /financialtimes/
Allow: /pages/
Allow: /2003/
Allow: /2004/
Allow: /2005/
Allow: /top/
Allow: /ref/
Allow: /services/xml/
 
User-agent: Mediapartners-Google*
Disallow:
 
# robots.txt, http://dictionary.reference.com
User-agent: Googlebot
Disallow:
 
User-agent: Mediapartners-Google
Disallow:
 
User-agent: Teleport Pro
Disallow: /
 
User-agent: *
Disallow: /cgi-bin/
 
# robots.txt for www.phpbbhacks.com
User-agent: *
Disallow: /forums/viewtopic.php
Disallow: /forums/viewforum.php
Disallow: /forums/index.php?
Disallow: /forums/posting.php
Disallow: /forums/groupcp.php
Disallow: /forums/search.php
Disallow: /forums/login.php
Disallow: /forums/privmsg.php
Disallow: /forums/post
Disallow: /forums/profile.php
Disallow: /forums/memberlist.php
Disallow: /forums/faq.php
Disallow: /forums/archive
 
# robots.txt for Slashdot.org
#
# "Any empty [Disallow] value, indicates that all URLs can be retrieved.
# At least one Disallow field needs to be present in a record."
 
User-agent: Mediapartners-Google
Disallow:
 
User-agent: Googlebot
Crawl-delay: 100
Disallow: /firehose.pl
Disallow: /submit.pl
Disallow: /comments.pl
Disallow: /users.pl
Disallow: /zoo.pl
Disallow: firehose.pl
Disallow: submit.pl
Disallow: comments.pl
Disallow: users.pl
Disallow: zoo.pl
Disallow: /~
Disallow: ~
 
User-agent: Slurp
Crawl-delay: 100
Disallow:
 
User-agent: Yahoo-NewsCrawler
Disallow:
 
User-Agent: msnbot
Crawl-delay: 100
Disallow:
 
User-agent: *
Crawl-delay: 100
Disallow: /authors.pl
Disallow: /index.pl
Disallow: /article.pl
Disallow: /comments.pl
Disallow: /firehose.pl
Disallow: /journal.pl
Disallow: /messages.pl
Disallow: /metamod.pl
Disallow: /users.pl
Disallow: /search.pl
Disallow: /submit.pl
Disallow: /pollBooth.pl
Disallow: /pubkey.pl
Disallow: /topics.pl
Disallow: /zoo.pl
Disallow: /palm
Disallow: authors.pl
Disallow: index.pl
Disallow: article.pl
Disallow: comments.pl
Disallow: firehose.pl
Disallow: journal.pl
Disallow: messages.pl
Disallow: metamod.pl
Disallow: users.pl
Disallow: search.pl
Disallow: submit.pl
Disallow: pollBooth.pl
Disallow: pubkey.pl
Disallow: topics.pl
Disallow: zoo.pl
Disallow: /~
Disallow: ~
 
# robots.txt for http://www.myspace.com
User-agent: ia_archiver
Disallow: /
 
# robots.txt for http://www.craigslist.com
User-agent: YahooFeedSeeker
Disallow: /forums
Disallow: /res/
Disallow: /post
Disallow: /email.friend
Disallow: /?flagCode
Disallow: /ccc
Disallow: /hhh
Disallow: /sss
Disallow: /bbb
Disallow: /ggg
Disallow: /jjj
 
User-agent: *
Disallow: /cgi-bin
Disallow: /cgi-secure
Disallow: /forums
Disallow: /search
Disallow: /res/
Disallow: /post
Disallow: /email.friend
Disallow: /?flagCode
Disallow: /ccc
Disallow: /hhh
Disallow: /sss
Disallow: /bbb
Disallow: /ggg
Disallow: /jjj
 
User-Agent: OmniExplorer_Bot
Disallow: /
 
# robots.txt for http://www.alexa.com
User-agent: googlebot  # allow Google crawler
Disallow: /search
 
User-agent: gulliver  # allow Northern Light crawler
Disallow: /search
 
User-agent: slurp  # allow Inktomi crawler
Disallow: /search
 
User-agent: fast  # allow FAST crawler
Disallow: /search
 
User-agent: scooter  # allow AltaVista crawler
Disallow: /search
 
User-agent: vscooter  # allow AltaVista image crawler
Disallow: /search
 
User-agent: ia_archiver  # allow Internet Archive crawler
Disallow: /search
 
User-agent: *    # Disallow all other crawlers access
Disallow: /
 
# robots.txt for http://www.technorati.com
User-agent: NPBot
Disallow: /
 
User-agent: TurnitinBot
Disallow: /
 
User-Agent: sitecheck.internetseer.com
Disallow: /
 
User-Agent: *
Crawl-Delay: 3
Disallow: /search/
Disallow: /search.php
Disallow: /cosmos.php
 
# robots.txt for www.sitepoint.com
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /forums/report.php
Disallow: /forums/search.php
Disallow: /forums/newreply.php
Disallow: /forums/editpost.php
Disallow: /forums/memberlist.php
Disallow: /forums/profile.php
Disallow: /launch/
Disallow: /search/
Disallow: /voucher/424/
Disallow: /email/
Disallow: /feedback/
Disallow: /contact?reason=articlesuggest
Disallow: /linktothis/
Disallow: /popup/
Disallow: /forums/archive/
 
# robots.txt for http://www.w3.org/
 
# For use by search.w3.org
User-agent: W3C-gsa
Disallow: /Out-Of-Date
 
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
 
# W3C Link checker
User-agent: W3C-checklink
Disallow:
 
# exclude some access-controlled areas
User-agent: *
Disallow: /2004/ontaria/basic
Disallow: /Team
Disallow: /Project
Disallow: /Systems
Disallow: /Web
Disallow: /History
Disallow: /Out-Of-Date
Disallow: /2002/02/mid
Disallow: /mid/
Disallow: /People/all/
Disallow: /RDF/Validator/ARPServlet
Disallow: /2003/03/Translations/byLanguage
Disallow: /2003/03/Translations/byTechnology
Disallow: /2005/11/Translations/Query
Disallow: /2003/glossary/subglossary/
#Disallow: /2005/06/blog/
#Disallow: /2001/07/pubrules-checker
#shouldnt get transparent proxies but will ml links of things like pubrules
Disallow: /2000/06/webdata/xslt
Disallow: /2000/09/webdata/xslt
Disallow: /2005/08/online_xslt/xslt
Disallow: /Bugs/
Disallow: /Search/Mail/Public/
Disallow: /2006/02/chartergen
 
# robots.txt for www.google-analytics.com
User-Agent: *
Disallow: /
Noindex: /
 
# robots.txt for video.google.com
User-agent: *
Disallow: /videosearch?
Disallow: /videofeed?
Disallow: /videopreview?
Disallow: /videopreviewbig?
Disallow: /videoprograminfo?
Disallow: /videorandom
Disallow: /videolineup
Disallow: /downloadgvp
 
# robots.txt for www.google.com
User-agent: *
Allow: /searchhistory/
Disallow: /news?output=xhtml&
Allow: /news?output=xhtml
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalogues
Disallow: /news
Disallow: /nwshp
Disallow: /?
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /sorry/
Disallow: /imgres
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /advanced_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /m?
Disallow: /m/search?
Disallow: /wml?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /froogle?
Disallow: /froogle_
Disallow: /print
Disallow: /books
Disallow: /patents?
Disallow: /scholar?
Disallow: /complete
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Disallow: /maps?
Disallow: /translate?
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /reader/
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /notebook/search?
Disallow: /music
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/search?
Disallow: /base/reportbadoffer
Disallow: /base/s2
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Disallow: /finance
Disallow: /reviews/search?
 
# robots.txt for validator.w3.org
# $Id: robots.txt,v 1.3 2000/12/13 13:04:09 gerald Exp $
 
User-agent: *
Disallow: /check
 
# robots.txt for httpd.apache.org
User-agent: *
Disallow: /websrc
 
# robots.txt for www.apache.org
User-agent: *
Disallow: /websrc
Crawl-Delay: 4

---

#  Please, we do NOT allow nonauthorized robots.
#  http://www.webmasterworld.com/robots
#  Actual robots can always be found here for: http://www.webmasterworld.com/robots2
#  Old full robots.txt can be found here: http://www.webmasterworld.com/robots3
#  Any unauthorized bot running will result in IP's being banned.
#  Agent spoofing is considered a bot.
#  Fair warning to the clueless - honey pots are - and have been - running.
#  If you have been banned for bot running - please sticky an admin for a reinclusion request.
#  http://www.searchengineworld.com/robots/
#  This code found here: http://www.webmasterworld.com/robots.txt?view=rawcode
 
User-agent: *
Crawl-delay: 17
 
User-agent: *
Disallow: /gfx/
Disallow: /cgi-bin/
Disallow: /QuickSand/
Disallow: /pda/
Disallow: /zForumFFFFFF/

# WebmasterWorld.com: robots.txt
# GNU Robots.txt Feel free to use with credit
# given to WebmasterWorld.
# Please, we do NOT allow nonauthorized robots any longer.
# http://www.searchengineworld.com/robots/
# Yes, feel free to copy and use the following.
 
User-agent: OmniExplorer_Bot
Disallow: /
 
User-agent: FreeFind
Disallow: /
 
User-agent: BecomeBot
Disallow: /
 
User-agent: Nutch
Disallow: /
 
User-agent: Jetbot/1.0
Disallow: /
 
User-agent: Jetbot
Disallow: /
 
User-agent: WebVac
Disallow: /
 
User-agent: Stanford
Disallow: /
 
User-agent: naver
Disallow: /
 
User-agent: dumbot
Disallow: /
 
User-agent: Hatena Antenna
Disallow: /
 
User-agent: grub-client
Disallow: /
 
User-agent: grub
Disallow: /
 
User-agent: looksmart
Disallow: /
 
User-agent: WebZip
Disallow: /
 
User-agent: larbin
Disallow: /
 
User-agent: b2w/0.1
Disallow: /
 
User-agent: Copernic
Disallow: /
 
User-agent: psbot
Disallow: /
 
User-agent: Python-urllib
Disallow: /
 
User-agent: Googlebot-Image
Disallow: /
 
User-agent: NetMechanic
Disallow: /
 
User-agent: URL_Spider_Pro
Disallow: /
 
User-agent: CherryPicker
Disallow: /
 
User-agent: EmailCollector
Disallow: /
 
User-agent: EmailSiphon
Disallow: /
 
User-agent: WebBandit
Disallow: /
 
User-agent: EmailWolf
Disallow: /
 
User-agent: ExtractorPro
Disallow: /
 
User-agent: CopyRightCheck
Disallow: /
 
User-agent: Crescent
Disallow: /
 
User-agent: SiteSnagger
Disallow: /
 
User-agent: ProWebWalker
Disallow: /
 
User-agent: CheeseBot
Disallow: /
 
User-agent: LNSpiderguy
Disallow: /
 
User-agent: Mozilla
Disallow: /
 
User-agent: mozilla
Disallow: /
 
User-agent: mozilla/3
Disallow: /
 
User-agent: mozilla/4
Disallow: /
 
User-agent: mozilla/5
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow: /
 
User-agent: ia_archiver
Disallow: /
 
User-agent: ia_archiver/1.6
Disallow: /
 
User-agent: Alexibot
Disallow: /
 
User-agent: Teleport
Disallow: /
 
User-agent: TeleportPro
Disallow: /
 
User-agent: Stanford Comp Sci
Disallow: /
 
User-agent: MIIxpc
Disallow: /
 
User-agent: Telesoft
Disallow: /
 
User-agent: Website Quester
Disallow: /
 
User-agent: moget/2.1
Disallow: /
 
User-agent: WebZip/4.0
Disallow: /
 
User-agent: WebStripper
Disallow: /
 
User-agent: WebSauger
Disallow: /
 
User-agent: WebCopier
Disallow: /
 
User-agent: NetAnts
Disallow: /
 
User-agent: Mister PiX
Disallow: /
 
User-agent: WebAuto
Disallow: /
 
User-agent: TheNomad
Disallow: /
 
User-agent: WWW-Collector-E
Disallow: /
 
User-agent: RMA
Disallow: /
 
User-agent: libWeb/clsHTTP
Disallow: /
 
User-agent: asterias
Disallow: /
 
User-agent: httplib
Disallow: /
 
User-agent: turingos
Disallow: /
 
User-agent: spanner
Disallow: /
 
User-agent: InfoNaviRobot
Disallow: /
 
User-agent: Harvest/1.5
Disallow: /
 
User-agent: Bullseye/1.0
Disallow: /
 
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /
 
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
 
User-agent: CherryPickerSE/1.0
Disallow: /
 
User-agent: CherryPickerElite/1.0
Disallow: /
 
User-agent: WebBandit/3.50
Disallow: /
 
User-agent: NICErsPRO
Disallow: /
 
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
 
User-agent: DittoSpyder
Disallow: /
 
User-agent: Foobot
Disallow: /
 
User-agent: WebmasterWorldForumBot
Disallow: /
 
User-agent: SpankBot
Disallow: /
 
User-agent: BotALot
Disallow: /
 
User-agent: lwp-trivial/1.34
Disallow: /
 
User-agent: lwp-trivial
Disallow: /
 
User-agent: http://www.WebmasterWorld.com bot
Disallow: /
 
User-agent: BunnySlippers
Disallow: /
 
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
 
User-agent: URLy Warning
Disallow: /
 
User-agent: Wget/1.6
Disallow: /
 
User-agent: Wget/1.5.3
Disallow: /
 
User-agent: Wget
Disallow: /
 
User-agent: LinkWalker
Disallow: /
 
User-agent: cosmos
Disallow: /
 
User-agent: moget
Disallow: /
 
User-agent: hloader
Disallow: /
 
User-agent: humanlinks
Disallow: /
 
User-agent: LinkextractorPro
Disallow: /
 
User-agent: Offline Explorer
Disallow: /
 
User-agent: Mata Hari
Disallow: /
 
User-agent: LexiBot
Disallow: /
 
User-agent: Web Image Collector
Disallow: /
 
User-agent: The Intraformant
Disallow: /
 
User-agent: True_Robot/1.0
Disallow: /
 
User-agent: True_Robot
Disallow: /
 
User-agent: BlowFish/1.0
Disallow: /
 
User-agent: http://www.SearchEngineWorld.com bot
Disallow: /
 
User-agent: http://www.WebmasterWorld.com bot
Disallow: /
 
User-agent: JennyBot
Disallow: /
 
User-agent: MIIxpc/4.2
Disallow: /
 
User-agent: BuiltBotTough
Disallow: /
 
User-agent: ProPowerBot/2.14
Disallow: /
 
User-agent: BackDoorBot/1.0
Disallow: /
 
User-agent: toCrawl/UrlDispatcher
Disallow: /
 
User-agent: WebEnhancer
Disallow: /
 
User-agent: suzuran
Disallow: /
 
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
 
User-agent: VCI
Disallow: /
 
User-agent: Szukacz/1.4
Disallow: /
 
User-agent: QueryN Metasearch
Disallow: /
 
User-agent: Openfind data gathere
Disallow: /
 
User-agent: Openfind
Disallow: /
 
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
 
User-agent: Xenu's
Disallow: /
 
User-agent: Zeus
Disallow: /
 
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
 
User-agent: RepoMonkey
Disallow: /
 
User-agent: Microsoft URL Control
Disallow: /
 
User-agent: Openbot
Disallow: /
 
User-agent: URL Control
Disallow: /
 
User-agent: Zeus Link Scout
Disallow: /
 
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
 
User-agent: Webster Pro
Disallow: /
 
User-agent: EroCrawler
Disallow: /
 
User-agent: LinkScan/8.1a Unix
Disallow: /
 
User-agent: Keyword Density/0.9
Disallow: /
 
User-agent: Kenjin Spider
Disallow: /
 
User-agent: Iron33/1.0.2
Disallow: /
 
User-agent: Bookmark search tool
Disallow: /
 
User-agent: GetRight/4.2
Disallow: /
 
User-agent: FairAd Client
Disallow: /
 
User-agent: Gaisbot
Disallow: /
 
User-agent: Aqua_Products
Disallow: /
 
User-agent: Radiation Retriever 1.1
Disallow: /
 
User-agent: WebmasterWorld Extractor
Disallow: /
 
User-agent: Flaming AttackBot
Disallow: /
 
User-agent: Oracle Ultra Search
Disallow: /
 
User-agent: MSIECrawler
Disallow: /
 
User-agent: PerMan
Disallow: /
 
User-agent: searchpreview
Disallow: /
 
User-agent: sootle
Disallow: /
 
User-agent: es
Disallow: /
 
User-agent: Enterprise_Search/1.0
Disallow: /
 
User-agent: Enterprise_Search
Disallow: /
 
User-agent: *
Disallow: /gfx/
Disallow: /cgi-bin/
Disallow: /QuickSand/
Disallow: /pda/
Disallow: /zForumFFFFFF/

---

You don't have to block your feeds from indexing. Matt Cutts himself suggested not to block those because there is not real reason to doing so. If you are talking about a blog and its /feed and such urls it won't cause mess in your rankings, so my suggesting would be not touch those feeds.

As for blocking, no you won't have any affection to your main url if you block /feed or whatever urls you want. Gbot will just deindex them and stop crawling them.

Wget Robot Exclusion

It is extremely easy to make Wget wander aimlessly around a web site, sucking all the available data in progress. `wget -r site', and you're set. Great? Not for the server admin.

As long as Wget is only retrieving static pages, and doing it at a reasonable rate (see the `--wait' option), there's not much of a problem. The trouble is that Wget can't tell the difference between the smallest static page and the most demanding CGI. A site I know has a section handled by an, uh, bitchin' CGI Perl script that converts Info files to HTML on the fly. The script is slow, but works well enough for human users viewing an occasional Info file. However, when someone's recursive Wget download stumbles upon the index page that links to all the Info files through the script, the system is brought to its knees without providing anything useful to the downloader.

To avoid this kind of accident, as well as to preserve privacy for documents that need to be protected from well-behaved robots, the concept of robot exclusion has been invented. The idea is that the server administrators and document authors can specify which portions of the site they wish to protect from the robots.

The most popular mechanism, and the de facto standard supported by all the major robots, is the "Robots Exclusion Standard" (RES) written by Martijn Koster et al. in 1994. It specifies the format of a text file containing directives that instruct the robots which URL paths to avoid. To be found by the robots, the specifications must be placed in `/robots.txt' in the server root, which the robots are supposed to download and parse.

Although Wget is not a web robot in the strictest sense of the word, it can downloads large parts of the site without the user's intervention to download an individual page. Because of that, Wget honors RES when downloading recursively. For instance, when you issue: wget -r http://www.server.com/ First the index of `www.server.com' will be downloaded. If Wget finds that it wants to download more documents from that server, it will request `http://www.server.com/robots.txt' and, if found, use it for further downloads. `robots.txt' is loaded only once per each server.

Until version 1.8, Wget supported the first version of the standard, written by Martijn Koster in 1994 and available at http://www.robotstxt.org/wc/norobots.html. As of version 1.8, Wget has supported the additional directives specified in the internet draft draft-koster-robots-00.txt titled "A Method for Web Robots Control". The draft, which has as far as I know never made to an RFC, is available at http://www.robotstxt.org/wc/norobots-rfc.txt.

This manual no longer includes the text of the Robot Exclusion Standard.

The second, less known mechanism, enables the author of an individual document to specify whether they want the links from the file to be followed by a robot. This is achieved using the META tag, like this: <meta name="robots" content="nofollow"> This is explained in some detail at http://www.robotstxt.org/wc/meta-user.html. Wget supports this method of robot exclusion in addition to the usual /robots.txt exclusion.

If you know what you are doing and really really wish to turn off the robot exclusion, set the robots variable to `off' in your `.wgetrc'. You can achieve the same effect from the command line using the -e switch, e.g. wget -e robots=off url

SEO with Robots.txt originally appeared on AskApache.com

]]> http://www.askapache.com/seo/seo-with-robotstxt.html/feed 22