HTTP Headers

9 Method Definitions

The set of common methods for HTTP/1.1 is defined below. Although this set can be expanded, additional methods cannot be assumed to share the same semantics for separately extended clients and servers.

The Host request-header field (section 14.23) MUST accompany all HTTP/1.1 requests.

9.1 Safe and Idempotent Methods

9.1.1 Safe Methods

Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they might take which may have an unexpected significance to themselves or others.

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.

9.1.2 Idempotent Methods

Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. Also, the methods OPTIONS and TRACE SHOULD NOT have side effects, and so are inherently idempotent.

However, it is possible that a sequence of several requests is non- idempotent, even if all of the methods executed in that sequence are idempotent. (A sequence is idempotent if a single execution of the entire sequence always yields a result that is not changed by a reexecution of all, or part, of that sequence.) For example, a sequence is non-idempotent if its result depends on a value that is later modified in the same sequence.

A sequence that never has side effects is idempotent, by definition (provided that no concurrent operations are being executed on the same set of resources).

9.2 OPTIONS

The OPTIONS method represents a request for information about the communication options available on the request/response chain identified by the Request-URI. This method allows the client to determine the options and/or requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval.

Responses to this method are not cacheable.

If the OPTIONS request includes an entity-body (as indicated by the presence of Content-Length or Transfer-Encoding), then the media type MUST be indicated by a Content-Type field. Although this specification does not define any use for such a body, future extensions to HTTP might use the OPTIONS body to make more detailed queries on the server. A server that does not support such an extension MAY discard the request body.

If the Request-URI is an asterisk ("*"), the OPTIONS request is intended to apply to the server in general rather than to a specific resource. Since a server's communication options typically depend on the resource, the "*" request is only useful as a "ping" or "no-op" type of method; it does nothing beyond allowing the client to test the capabilities of the server. For example, this can be used to test a proxy for HTTP/1.1 compliance (or lack thereof).

If the Request-URI is not an asterisk, the OPTIONS request applies only to the options that are available when communicating with that resource.

A 200 response SHOULD include any header fields that indicate optional features implemented by the server and applicable to that resource (e.g., Allow), possibly including extensions not defined by this specification. The response body, if any, SHOULD also include information about the communication options. The format for such a

body is not defined by this specification, but might be defined by future extensions to HTTP. Content negotiation MAY be used to select the appropriate response format. If no response body is included, the response MUST include a Content-Length field with a field-value of "0".

The Max-Forwards request-header field MAY be used to target a specific proxy in the request chain. When a proxy receives an OPTIONS request on an absoluteURI for which request forwarding is permitted, the proxy MUST check for a Max-Forwards field. If the Max-Forwards field-value is zero ("0"), the proxy MUST NOT forward the message; instead, the proxy SHOULD respond with its own communication options. If the Max-Forwards field-value is an integer greater than zero, the proxy MUST decrement the field-value when it forwards the request. If no Max-Forwards field is present in the request, then the forwarded request MUST NOT include a Max-Forwards field.

9.3 GET

The GET method means retrieve whatever information (in the form of an entity) is identified by the Request-URI. If the Request-URI refers to a data-producing process, it is the produced data which shall be returned as the entity in the response and not the source text of the process, unless that text happens to be the output of the process.

The semantics of the GET method change to a "conditional GET" if the request message includes an If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match, or If-Range header field. A conditional GET method requests that the entity be transferred only under the circumstances described by the conditional header field(s). The conditional GET method is intended to reduce unnecessary network usage by allowing cached entities to be refreshed without requiring multiple requests or transferring data already held by the client.

The semantics of the GET method change to a "partial GET" if the request message includes a Range header field. A partial GET requests that only part of the entity be transferred, as described in section 14.35. The partial GET method is intended to reduce unnecessary network usage by allowing partially-retrieved entities to be completed without transferring data already held by the client.

The response to a GET request is cacheable if and only if it meets the requirements for HTTP caching described in section 13.

See section 15.1.3 for security considerations when used for forms.

9.4 HEAD

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.

The response to a HEAD request MAY be cacheable in the sense that the information contained in the response MAY be used to update a previously cached entity from that resource. If the new field values indicate that the cached entity differs from the current entity (as would be indicated by a change in Content-Length, Content-MD5, ETag or Last-Modified), then the cache MUST treat the cache entry as stale.

9.5 POST

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line. POST is designed to allow a uniform method to cover the following functions:

      - Annotation of existing resources; 

      - Posting a message to a bulletin board, newsgroup, mailing list, or similar group of articles; 

      - Providing a block of data, such as the result of submitting a form, to a data-handling process; 

      - Extending a database through an append operation. 

The actual function performed by the POST method is determined by the server and is usually dependent on the Request-URI. The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database.

The action performed by the POST method might not result in a resource that can be identified by a URI. In this case, either 200 (OK) or 204 (No Content) is the appropriate response status, depending on whether or not the response includes an entity that describes the result.

If a resource has been created on the origin server, the response SHOULD be 201 (Created) and contain an entity which describes the status of the request and refers to the new resource, and a Location header (see section 14.30).

Responses to this method are not cacheable, unless the response includes appropriate Cache-Control or Expires header fields. However, the 303 (See Other) response can be used to direct the user agent to retrieve a cacheable resource.

POST requests MUST obey the message transmission requirements set out in section 8.2.

See section 15.1.3 for security considerations.

9.6 PUT

The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI. If a new resource is created, the origin server MUST inform the user agent via the 201 (Created) response. If an existing resource is modified, either the 200 (OK) or 204 (No Content) response codes SHOULD be sent to indicate successful completion of the request. If the resource could not be created or modified with the Request-URI, an appropriate error response SHOULD be given that reflects the nature of the problem. The recipient of the entity MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501 (Not Implemented) response in such cases.

If the request passes through a cache and the Request-URI identifies one or more currently cached entities, those entries SHOULD be treated as stale. Responses to this method are not cacheable.

The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource. If the server desires that the request be applied to a different URI,

it MUST send a 301 (Moved Permanently) response; the user agent MAY then make its own decision regarding whether or not to redirect the request.

A single resource MAY be identified by many different URIs. For example, an article might have a URI for identifying "the current version" which is separate from the URI identifying each particular version. In this case, a PUT request on a general URI might result in several other URIs being defined by the origin server.

HTTP/1.1 does not define how a PUT method affects the state of an origin server.

PUT requests MUST obey the message transmission requirements set out in section 8.2.

Unless otherwise specified for a particular entity-header, the entity-headers in the PUT request SHOULD be applied to the resource created or modified by the PUT.

9.7 DELETE

The DELETE method requests that the origin server delete the resource identified by the Request-URI. This method MAY be overridden by human intervention (or other means) on the origin server. The client cannot be guaranteed that the operation has been carried out, even if the status code returned from the origin server indicates that the action has been completed successfully. However, the server SHOULD NOT indicate success unless, at the time the response is given, it intends to delete the resource or move it to an inaccessible location.

A successful response SHOULD be 200 (OK) if the response includes an entity describing the status, 202 (Accepted) if the action has not yet been enacted, or 204 (No Content) if the action has been enacted but the response does not include an entity.

If the request passes through a cache and the Request-URI identifies one or more currently cached entities, those entries SHOULD be treated as stale. Responses to this method are not cacheable.

9.8 TRACE

The TRACE method is used to invoke a remote, application-layer loop- back of the request message. The final recipient of the request SHOULD reflect the message received back to the client as the entity-body of a 200 (OK) response. The final recipient is either the

origin server or the first proxy or gateway to receive a Max-Forwards value of zero (0) in the request (see section 14.31). A TRACE request MUST NOT include an entity.

TRACE allows the client to see what is being received at the other end of the request chain and use that data for testing or diagnostic information. The value of the Via header field (section 14.45) is of particular interest, since it acts as a trace of the request chain. Use of the Max-Forwards header field allows the client to limit the length of the request chain, which is useful for testing a chain of proxies forwarding messages in an infinite loop.

If the request is valid, the response SHOULD contain the entire request message in the entity-body, with a Content-Type of "message/http". Responses to this method MUST NOT be cached.

9.9 CONNECT

This specification reserves the method name CONNECT for use with a proxy that can dynamically switch to being a tunnel (e.g. SSL tunneling [44]).

List of HTTP Response Status Codes

1xx Info / Informational

HTTP_INFO - Request received, continuing process.

Indicates a provisional response, consisting only of the Status-Line and optional headers, and is terminated by an empty line.

• 100 Continue - HTTP_CONTINUE
• 101 Switching Protocols - HTTP_SWITCHING_PROTOCOLS
• 102 Processing - HTTP_PROCESSING

2xx Success / OK

HTTP_SUCCESS - The action was successfully received, understood, and accepted.

Indicates that the client's request was successfully received, understood, and accepted.

• 200 OK - HTTP_OK
• 201 Created - HTTP_CREATED
• 202 Accepted - HTTP_ACCEPTED
• 203 Non-Authoritative Information - HTTP_NON_AUTHORITATIVE
• 204 No Content - HTTP_NO_CONTENT
• 205 Reset Content - HTTP_RESET_CONTENT
• 206 Partial Content - HTTP_PARTIAL_CONTENT
• 207 Multi-Status - HTTP_MULTI_STATUS

3xx Redirect

HTTP_REDIRECT - The client must take additional action to complete the request.

Indicates that further action needs to be taken by the user-agent in order to fulfill the request. The action required may be carried out by the user agent without interaction with the user if and only if the method used in the second request is GET or HEAD. A user agent should not automatically redirect a request more than 5 times, since such redirections usually indicate an infinite loop.

• 300 Multiple Choices - HTTP_MULTIPLE_CHOICES
• 301 Moved Permanently - HTTP_MOVED_PERMANENTLY
• 302 Found - HTTP_MOVED_TEMPORARILY
• 303 See Other - HTTP_SEE_OTHER
• 304 Not Modified - HTTP_NOT_MODIFIED
• 305 Use Proxy - HTTP_USE_PROXY
• 306 unused - UNUSED
• 307 Temporary Redirect - HTTP_TEMPORARY_REDIRECT

4xx Client Error

HTTP_CLIENT_ERROR - The request contains bad syntax or cannot be fulfilled.

Indicates case where client seems to have erred. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition.

• 400 Bad Request - HTTP_BAD_REQUEST
• 401 Authorization Required - HTTP_UNAUTHORIZED
• 402 Payment Required - HTTP_PAYMENT_REQUIRED
• 403 Forbidden - HTTP_FORBIDDEN
• 404 Not Found - HTTP_NOT_FOUND
• 405 Method Not Allowed - HTTP_METHOD_NOT_ALLOWED
• 406 Not Acceptable - HTTP_NOT_ACCEPTABLE
• 407 Proxy Authentication Required - HTTP_PROXY_AUTHENTICATION_REQUIRED
• 408 Request Time-out - HTTP_REQUEST_TIME_OUT
• 409 Conflict - HTTP_CONFLICT
• 410 Gone - HTTP_GONE
• 411 Length Required - HTTP_LENGTH_REQUIRED
• 412 Precondition Failed - HTTP_PRECONDITION_FAILED
• 413 Request Entity Too Large - HTTP_REQUEST_ENTITY_TOO_LARGE
• 414 Request-URI Too Large - HTTP_REQUEST_URI_TOO_LARGE
• 415 Unsupported Media Type - HTTP_UNSUPPORTED_MEDIA_TYPE
• 416 Requested Range Not Satisfiable - HTTP_RANGE_NOT_SATISFIABLE
• 417 Expectation Failed - HTTP_EXPECTATION_FAILED
• 418 unused - UNUSED
• 419 unused - UNUSED
• 420 unused - UNUSED
• 421 unused - UNUSED
• 422 Unprocessable Entity - HTTP_UNPROCESSABLE_ENTITY
• 423 Locked - HTTP_LOCKED
• 424 Failed Dependency - HTTP_FAILED_DEPENDENCY
• 425 No code - HTTP_NO_CODE
• 426 Upgrade Required - HTTP_UPGRADE_REQUIRED

5xx Server Error

HTTP_SERVER_ERROR - The server failed to fulfill an apparently valid request.

Indicate cases in which the server is aware that it has erred or is incapable of performing the request. Except when responding to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. User agents should display any included entity to the user. These response codes are applicable to any request method.

• 500 Internal Server Error - HTTP_INTERNAL_SERVER_ERROR
• 501 Method Not Implemented - HTTP_NOT_IMPLEMENTED
• 502 Bad Gateway - HTTP_BAD_GATEWAY
• 503 Service Temporarily Unavailable - HTTP_SERVICE_UNAVAILABLE
• 504 Gateway Time-out - HTTP_GATEWAY_TIME_OUT
• 505 HTTP Version Not Supported - HTTP_VERSION_NOT_SUPPORTED
• 506 Variant Also Negotiates - HTTP_VARIANT_ALSO_VARIES
• 507 Insufficient Storage - HTTP_INSUFFICIENT_STORAGE
• 508 unused - UNUSED
• 509 unused - UNUSED
• 510 Not Extended - HTTP_NOT_EXTENDED

Helpful HTTP Links

1. HTTP specification, Section 10
2. TLS Upgrade within HTTP specification, Section 4
3. HTTP Status Code Registry
4. WebDAV specification, Section 11
5. IANA registry
6. Adobe Flash status code definitions (ie 408)
7. Microsoft Internet Information Server Status Codes and Sub-Codes
8. httplint
9. HTTP Headers, brief intro.
10. Common User-Agent Issues

DRP

"The HTTP Distribution and Replication Protocol"

DupSup

"Duplicate Suppression in HTTP"

EARL Schema

"Evaluation and Report Language (EARL) 1.0 Schema"

EDD

"An exploration of dynamic documents"

EdgeArch

"Edge Architecture Specification"

HttpClient

Jakarta Commons HttpClient

HTML4

"HTML 4.01 Specification"

JEPI

"White Paper: Joint Electronic Payment Initiative"

ObjectHeaders

"Object Header lines in HTTP"

OPS-OverHTTP

"Implementation of OPS Over HTTP"

OPTIONS messages

"Specification of HTTP/1.1 OPTIONS messages"

P3P

"The Platform for Privacy Preferences 1.0 (P3P1.0) Specification"

PEP

"PEP - an Extension Mechanism for HTTP"

PICSLabels

"PICS Label Distribution Label Syntax and Communication Protocols, Version 1.1"

Proxy Notification

"Notification for Proxy Caches"

RDF

"Resource Description Framework (RDF): Concepts and Abstract Syntax"

RDF-PRIMER

RDF Primer

RFC2068

"Hypertext Transfer Protocol -- HTTP/1.1"

RFC2109

"HTTP State Management Mechanism"

RFC2183

"Communicating Presentation Information in Internet Messages: The Content-Disposition Header Field"

RFC2227

"Simple Hit-Metering and Usage-Limiting for HTTP"

RFC2295

"Transparent Content Negotiation in HTTP"

RFC2310

"The Safe Response Header Field"

RFC2324

"Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)"

RFC2397

"The 'data' URL scheme"

RFC2518

"HTTP Extensions for Distributed Authoring -- WEBDAV"

RFC2616

"Hypertext Transfer Protocol -- HTTP/1.1"

RFC2617

"HTTP Authentication: Basic and Digest Access Authentication"

RFC2660

"The Secure HyperText Transfer Protocol"

RFC2774

"An HTTP Extension Framework"

RFC2965

"HTTP State Management Mechanism"

RFC3229

"Delta encoding in HTTP"

RFC3230

"Instance Digests in HTTP"

RFC3253

"Versioning Extensions to WebDAV (Web Distributed Authoring and Versioning)"

RFC3648

"Web Distributed Authoring and Versioning (WebDAV) Ordered Collections Protocol"

RFC3986

"Uniform Resource Identifier (URI): Generic Syntax"

RFC4229

"HTTP Header Field Registrations"

SOAP1.1

"Simple Object Access Protocol (SOAP) 1.1"

UA Attributes

"User-Agent Display Attributes Headers"

WIRE

"WIRE - W3 Identifier Resolution Extensions"

Raw HTTP Header Debugger originally appeared on AskApache.com

Advanced Web Development by AskApache is a Firefox Collection I created for myself to make finding and installing my favorite Firefox Add-ons simple and easy.

My Setup

As this screenshot shows, I only use a handful of add-ons at a time. These buttons allow me to clear the DNS/Cookies/Cache for whichever site I'm on when I click it. Very very helpful for me as a web designer. The 4th button there is just a restart button. Other than those, Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.

I like the idea of the last.fm but it's not as powerful as the site, which is awesome. Lately I've been listening to Kings of Leon Radio...

Slow Firefox

The more add-ons you have, disabled or not, the slower Firefox is. (unless you are running your profile folder in TMPFS). Also, bookmarks and settings like that have a performance hit. I have been using Firefox since it launched way back when, and I have always kept my bookmarks when moving to new machines and new installations.. So with over 5 thousand bookmarks I finally did some debugging and discovered that was a huge cause of Firefox acting slow. Now I am trying to migrate them all over to Google's Gmarks, which knowing Google will be awesome eventually.

Multiple Firefox Installations, Sorta

The solution to all these problems is to use separate Firefox profiles which use separate folders to store your profile-specific extensions and settings. So I have profiles with upwards of 40 Addons installed and enabled, and another profile that is built for speed... It's very very slow to be running Firebug and have multiple tabs open.. You can use the profilemanager to load them specific profiles from the command line. I personally use separate icons on my Windows Quick Launch that I just modified the shortcut pointing to firefox to also have the profile commandline. Note also that you can have multiple profiles open and running simultaneously.. This lets you do some crazy networking and other random hacks like having many Firefox instances running each of which is configured to use a separate SOCKS Proxy or network interface, so you can really open up those pipes for some intense txrx.

You can do some very powerful things with Firefox that most people are unaware of, if you are interested start with these:

• Setting up an extension development environment
• Firefox Command Line Options

About This Collection

Web Development Add-ons for Advanced Web Developers. I personally use these to work on Sites, Servers, WordPress, Javascript, PHP, CSS, XHTML, validation, page-loading, SEO, optimizing, and much more. Add this collection.

Created by:AskApache

Updated:September 22, 2009

Share this Collection

• Digg this!
• Post to Facebook
• Add to Delicious
• Post to MySpace
• Share on FriendFeed
• Post to Twitter

Firefox Add-ons for Web Developers originally appeared on AskApache.com

In part I: SEO Secrets of AskApache.com we talked about content and building a website to be your "SEO Base". This article discusses some advanced SEO concepts to get a site indexed, move your best pages higher in search results, and controlling the pagerank/seo-juice of your site. but relatively easy ways to control and tweak WHAT urls on your site are indexed, and HOW. I've heard some people refer to this as "controlling pagerank flow" and "controlling pagerank juice", basically we want our best pages to rank higher in the search engine results.

Big Picture: Going from no website to AskApache.com in less than a year can be accomplished by anyone with unique content and a resolve to avoid any shortcuts and take it one step at a time.

We want what Google wants, to provide the most relevant content for someone who is doing a search. Basically, you want every url on your site that has unique content to be included in the index. In other words, you have to think like a search engine!

The Goal of Google

Here's what I mean, Google approaches search with the overwhelming goal of bringing the content to a searcher that is the most-likely to be what that searcher is searching for. Another way of looking at it is something I read on Google...

Google's goal is to get you off of their site as fast as possible by providing you with exactly what you are looking for.

Think about SEO like this

If you search for htaccess tutorial for seo on Google, would you be more likely to visit a tutorial about using .htaccess for seo or a category page for htaccess articles? AskApache.com has both of those urls included in the index, but the article ranks higher than the category page, as it very well should.

Get Your URLS Indexed

Before I explain how I am able to help Google and other search engines rank my article pages higher than my category pages, we need to get the urls in the index or nothing will show up. There are many well-discussed methods for getting included in the index, so I'll just list a few that I use.

1. Provide High-Quality original content, people will link to it
2. Get a sitemap and use Google Webmaster Tools
3. Publish an RSS/Atom Feed and ping the net when you publish a post
4. Use robots.txt and robots meta tags

Your URLs in the index

Here's how to find out which of your pages are indexed.

• Indexed pages in your site: site:
• Pages that link to your site's front page: link:
• The current cache of your site: cache:
• Information about your site: info:
• Pages that are similar to your site: related:

Break It Down

Yo homeslice! I didn't mean break dance.. I mean lets simplify AskApache in the context of getting our urls indexed high/low. Here's the stats: 1 Homepage, 206 Articles, 19 Pages, 31 Categories

1 Homepage

This page is generally the highest ranking page in the index, it should contain links to your best urls, and provide easy navigation

206 Specific Topic Articles

These are the article's (like this one) of AskApache.com and are the main source of search engine traffic. You want each url (if its a good article) to be ranked as high as possible. Some keys are to really make each article specific to a topic by using best-practice (X)HTML.

19 Static Pages

Most of these are pages like the online-tools hosted on this site, or other basic pages like about, contact us, site-map, etc.. Some of these you may want to rank very high ( like the /about/ page ) and some you may not want to even be included in the index.

31 Specific Topic Category Pages

These are tricky because they are generally just lists of articles from each category, which isn't specific enough to get much seach-engine-traffic, but is very useful to site visitors. I beefed up my category pages by adding additional information about the category topic in addition to excerpts of the articles.

Higher Pagerank = Higher Up in Search Results

So Googlebot and other search engine robots have these crazy complicated algorithms (many patented) that SEO Industry types may get caught up in and try to technically analyze them. I'm sure you've seen/read/heard the complicated advice that will always be pushed by many... advice like:

• analyze the number of words in the description/title/1st paragraph/etc.
• Make sure your "keyword" is sprinkled throughout the text every 10-30th word..
• Other equally unexciting technical analysis

Now if you've had success with that then props to you, success is success, but I personally choose to completely ignore all that. The number 1 thing that the top search engines advise is to design your page for a Human Visitor, not a computer. The golden rule for me is how I would rank the page, not how some algorithm would.

Designing for a Human Visitor

This is a major factor in your site being at the top vs. nowhere. You design your HTML to be as minimal as possible (see source code for my homepage) and contain ONLY the neccessary elements. Above all, use semantically sound XHTML markup. (view source of W3C)

External CSS/Javascript

Get your javascript and CSS out of your HTML and use external files (like this site) ALWAYS! You should start with just the HTML, no css, no colors, no javascript, and THEN you add the .css and then you add the javascript.

Site Accessibility

Say your browser didn't have a mouse, didn't support images, css, javascript, or even colors! Your HTML should be structured such that your page is still easily readable and easy to navigate. You can use lynx, links, and many other terminal-based browsers to test for this... please see the Web Accessibility Initiative (WAI) for detailed info.

Web accessibility refers to the practice of making websites usable by people of all abilities and disabilities. When sites are correctly designed, developed and edited, all users can have equal access to information and functionality. For example, when a site is coded with semantically meaningful HTML, with textual equivalents provided for images and with links named meaningfully, this helps blind users using text-to-speech software and/or text-to-Braille hardware.

Controlling a URL's Pagerank

A few tools and techniques are available for controlling the "juice" or "pagerank" of your urls.

1. Robots.txt
2. Robots Meta Tags
3. Links

Robots.txt

I've done quite a bit of research and experimentation with robots.txt files, which is a file located in the root of your website at http://www.askapache.com/robots.txt that is downloaded by all legitimate search engine spiders/bots and used as a Blacklist to prevent certain urls from being indexed. Here are a few of the articles on this site, which you may skip if you like as they don't illustrate the big-picture that I am going to discuss now.

• Control Flow of Pagerank with robots.txt and NoFollow, NoIndex
• WordPress robots.txt
• SEO with Robots.txt
• Google AdSense using robots.txt
• WordPress robots.txt file optimized for SEO and Google

How To Use Robots.txt

Even though robots.txt files are for whitelisting and blacklisting urls, I have found that they should only be used as an extreme form of blacklisting. When you Disallow a url in your robots.txt file, that means most search engine bots won't even LOOK at the url. As you can see in the below example, I only disallow urls that shouldn't ever be LOOKED at. The real powertool is the robots meta tag.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
 
Sitemap: http://www.askapache.com/sitemap.xml

Robots Meta Tag

Ok I'm really trying to simplify, because what you should understand is the big-picture. Every page can have a robots meta tag in the header, and this robots meta tag can tell the search-engine to index/not-index AND follow/not-follow. Here are some examples:

<meta name="robots" content="index" />
<meta name="robots" content="noindex" />
<meta name="robots" content="noindex,follow" />
<meta name="robots" content="index,nofollow" />

content="index" / noindex

index means the search engine is free to index, archive, cache, and follow the page whereas noindex means DO NOT include this page in the search engine results.

content="follow" / nofollow

follow means the search engine is free to LOOK at the page and follow the links on the page whereas nofollow means DO NOT follow the links on the page.

WordPress Auto-Robots meta tag code

Just add this to any plugin file and it will add the right robots meta tag to your site.. tweak to taste.

function askapache_robots_header(){
 global $wpdb;
 
 $robot = '<meta name="robots" content="noindex,follow,nocache,noarchive" />';
 
 if ( is_paged() || is_search() || is_404() || is_author() || is_tag() )
   $robot = '<meta name="robots" content="noindex,follow" />';
 elseif ( is_home() || is_front_page() || is_single() )
   $robot = '<meta name="robots" content="follow,index" />';
 elseif ( is_category() || is_page() )
   $robot = '<meta name="robots" content="follow" />';
 
 echo $robot . "\n";
}
add_action( 'wp_head', 'askapache_robots_header' );

Links

External and Internal Links are the crux of SEO. It's important to start FIRST on your Internal Links and linking structure... Once you are satisfied that the correct pages are indexed and ranked appropriately, then you can begin to look at external links.

• The fewer links on a page, generally the better.
• If every page of your site points to the same url on your site, pagerank goes up from the number of internal links.
• Use of the rel, title, alt attribute semantically is very helpful. (next, prev, index)
• Put your best links higher up in the XHTML, and put helpful/solid links at the end.
• You can add rel="nofollow" to links that you dont want followed.

The web has gotten to be so full of malicious/non-helpful SEO activity that I recommend developing your content NOT external links. If you want to do this right and provide great content that makes search engine users happy and makes the web better, then explore this blog and develop content until the next article in this series, where I'll show you how to make your site explode.

Stay tuned for Part III, which will dive deeper into the pipeworks of AskApache.com

SEO Secrets of AskApache Part 2 originally appeared on AskApache.com

From pages on the W3C Quality Assurance

APIs

Application Program Interfaces

CC/PP

Composite Capability/Preference Profiles

CSS

Cascading Style Sheets

CSS

Cascading Style Sheets

DOM

Document Object Model

ERCIM

European Research Consortium for Informatics and Mathematics

GRDDL

Gleaning Resource Descriptions from Dialects of Languages

HTML

HyperText Markup Language

HTTP

Hypertext Transfer Protocol

MIT

Massachusetts Institute of Technology

OWL

OWL Web Ontology Language

P3P

Platform for Privacy Preferences

PICS

Platform for Internet Content Selection

PNG

Portable Network Graphics

POWDER

Protocol for Web Description Resources

QA

Quality Assurance

RDF

Resource Description Framework

RSS channel

RDF Site Summary

SMIL

Synchronized Multimedia Integration Language

SML

Service Modeling Language

SOAP

Soap

SPARQL

Simple Protocol and RDF Query Language

SVG

Scalable Vector Graphics

TAG

Technical Architecture Group

URI/URL

Uniform Resource Identifiers

W3C

World Wide Web Consortium

WAI

Web Accessibility Initiative

WS

Web Services

WS-CDL

Web Services Choreography Description Language

WSDL

Web Services Description Language

WebCGM

Web Computer Graphics Metafile

XForms

Next Generation Web Forms

XHTML

Extensible HyperText Markup Language

XHTML2

Extensible HyperText Markup Language

XLink

XML Link

XMLP

XML Protocol

XSL

Extensible Stylesheet Language

XML

Extensible Markup Language

XSLT

XSL Transformations

<abbr title="Application Program Interfaces">APIs</abbr>
<abbr title="Composite Capability/Preference Profiles">CC/PP</abbr>
<abbr title="Cascading Style Sheets">CSS</abbr>
<acronym title="Document Object Model">DOM</acronym>
<acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym>
<acronym title="Gleaning Resource Descriptions from Dialects of Languages">GRDDL</acronym>
<abbr title="HyperText Markup Language">HTML</abbr>
<acronym title="Massachusetts Institute of Technology">MIT</acronym>
<acronym title="OWL Web Ontology Language">OWL</acronym>
<abbr title="Platform for Privacy Preferences">P3P</abbr>
<acronym title="Platform for Internet Content Selection">PICS</acronym>
<acronym title="Portable Network Graphics">PNG</acronym>
<acronym title="Protocol for Web Description Resources">POWDER</acronym>
<abbr title="Quality Assurance">QA</abbr>
<abbr title="Resource Description Framework">RDF</abbr>
<abbr title="RDF Site Summary">RSS channel</abbr>
<acronym title="Synchronized Multimedia Integration Language">SMIL</acronym>
<abbr title="Service Modeling Language">SML</abbr>
<acronym title="Soap">SOAP</acronym>
<acronym title="Simple Protocol and RDF Query Language">SPARQL</acronym>
<abbr title="Scalable Vector Graphics">SVG</abbr>
<acronym title="Technical Architecture Group">TAG</acronym>
<acronym title="World Wide Web Consortium">W3C</acronym>
<abbr title="Uniform Resource Identifiers">URI/URL</abbr>
<acronym title="Web Accessibility Initiative">WAI</acronym>
<abbr title="Web Services Choreography Description Language">WS-CDL</abbr>
<abbr title="Extensible Markup Language">XML</abbr>
<acronym title="Web Services Description Language">WSDL</acronym>
<abbr title="Web Computer Graphics Metafile">WebCGM</abbr>
<acronym title="Next Generation Web Forms">XForms</acronym>
<abbr title="Extensible HyperText Markup Language">XHTML</abbr>
<abbr title="Extensible HyperText Markup Language">XHTML2</abbr>
<acronym title="XML Link">XLink</acronym>
<abbr title="XML Protocol">XMLP</abbr>
<acronym title="Extensible Stylesheet Language">XSL</acronym>
<acronym title="XSL Transformations">XSLT</acronym>

Abbr and Acronym examples originally appeared on AskApache.com

Web Design Process

Let me break down the process

XHTML == structure CSS == presentation CONTENT == content SCRIPTS == dynamic structure/presentation/content

Web Programming Languages

Optimizing for speed and accessiblity

Traffic shaping

Web Design Conclusion

Best-practice Web Design Process originally appeared on AskApache.com

WordPress as a blogging platform is ideal when it comes to SEO, the default setup is optimized for search engines.

Its features and functions guide a search engine through the posts, pages, and categories to help the search engine crawl your site and gather the information it needs to include your site within its database.

Check out: robots.txt

Top 10 SEO Factors

1. Title Tag - 4.57
2. Anchor Text of Links - 4.46
3. Keyword Use in Document Text - 4.38
4. Accessibility of Document - 4.3
5. Links to Document from Site-Internal Pages - 4.15
6. Primary Subject Matter of Site - 4.00
7. External Links to Linking Pages - 3.92
8. Link Popularity of Site in Topical Community - 3.77
9. Global Link Popularity of Site - 3.69
10. Keyword Spamming - 3.69

WordPress comes with several built in search optimization tools, including the ability to use .htaccess to create apparently static URLs called permalinks, blogrolling, and pinging. There are also a number of third party plugins and hacks which can be used for search engine optimization (SEO).

However, once you start using various WordPress Themes and customizing WordPress to meet your own needs, you may break some of those useful search engine friendly features. To maintain your WordPress site's optimal friendliness towards search engine spiders and crawlers, here are a few tips:

robots.txt usage for SEO boost
Use XHTML Phrase elements for SEO boost

XHTML SEO friendly elements

EM

Indicates emphasis.

STRONG

Indicates stronger emphasis.

CITE

Contains a citation or a reference to other sources.

DFN

Indicates that this is the defining instance of the enclosed term.

CODE

Designates a fragment of computer code.

SAMP

Designates sample output from programs, scripts, etc.

KBD

Indicates text to be entered by the user.

VAR

Indicates an instance of a variable or program argument.

abbr

Indicates an abbreviated form (e.g., WWW, HTTP, URI, Mass., etc.).

acronym

Indicates an acronym (e.g., WAC, radar, etc.).

• Good, Clean Code
  Make sure your site's code validates. Errors in your code may prevent a search engine from moving through the site successfully.
• Content Talks
  Search engines can't "see" a site. They can only "read" a site. Pretty does not talk to a search engine. What "talks" to a search engine are the words, the content, the material in your site that explains, shares, informs, educates, and babbles. Make sure you have quality word content for a search engine to examine and compare with all the parts and pieces to give you a good "score".
• Write Your Content with Searchers in Mind
  How do you find information on the Internet? If you are writing something that you want to be "found" on the Internet, think about the words and phrases someone would use to find your information. Use them more than once as you write, but not in every sentence. Learn how search engines scan your content, evaluate it, and categorize it so you can help yourself get in good favor with search engines.
• Content First
  A search engine enters your site and, for the most part, ignores the styles and CSS. It just plows through the site gathering content and information. Most WordPress Themes are designed with the content as close to the top of the unstyled page as possible, keeping sidebars and footers towards the bottom. Few search engines scan more than the first third of the page before moving on. Make sure your Theme puts the content near the top.
• Keywords, Links, and Titles Meet Content
  Search engines do not evaluate your site on how pretty it is, but they do evaluate the words and put them through a sifter, giving credit to certain words and combinations of words. Words found within your meta tag keywords listings and within your document are compared to words found within your links and titles. The more that match, the better your "score."
• Content in Links and Images
  Your site may not have much text, mostly photographs and links, but you have places in which to add textual content. Search engines look for alt and title in link and image tags. While these have a bigger purpose of making your site more accessible, having good descriptions and words in these attributes helps provide more content for search engines to digest.

• Link Popularity
  It is not how good your site is, it is how good the sites are that link to you. This still holds weight with search engine favoritism. It's about who links to you. Blogrolls, pingbacks, and trackbacks are all built into WordPress. These help you link to other people, which gives them credit, but it also helps them link to you, connecting the "links." The number of incoming links your site has that have been recognized by Google can be checked by typing link:www.yoursite.com into Google (other search engines have similar functions). Other ways to generate incomming links to your site include:

  • Add your site's url to your signature on forum posts on other sites.
  • Submit your site to directories (see below).
  • Note: Leaving comments on blogs will not help with this, since all modern blogging tools use the rel="nofollow" attribute. Don't be a comment spammer.

• Good Navigation Links
  A search engine crawls through your site, moving from page to page. Good navigational links to the categories, archives, and various pages on your site will invite a search engine to move gracefully from one page to another, following the connecting links and visiting most of your site.

Search Engine Site Submissions

There are many resources that will "help" you submit your site to search engines. Some are free, some for a fee. Or you can manually submit your site to search engines yourself. Whatever method you choose to use, once your site has been checked for errors and is ready to go, search engines will welcome your WordPress site.

Here are some tips for successful site submissions:

• Make sure you have content for search engines to scan. In general, have more than 10 posts on your site to give the search engines something to examine and evaluate.
• Do not submit your site to the same search engine more than once a month or longer, depending upon their criteria, not your anxiousness to be listed.
• Have ready to type, or copy and paste, a description of your site that is less than 200 words long, the title of the site, and the categories your site may belong to in a search engine directory.
• Have a list of your website's various "addresses/URLs" ready. You can submit your root directory as well as specific categories and feeds to search engines, expanding your search engine coverage.
• Keep a list of the various search engines and directories you submit to so you do not accidentally resubmit too soon, and you can keep track of how they include you among their pages and results.

Directory Sites

It is also useful for traffic generation and search optimization purposes to submit your site to directories. Both comprehensive directory sites and those specific to the subject or localisation of your site can be used.

DMOZ.org(http://www.dmoz.org) this is the most important directory - it's content is licensed in an open fashion allowing it to be syndicated through out the web -- its content is also used directly in some fashion by almost all of the major search engines.

Search Engine Optimization Resources

While WordPress comes ready for search engines, the following are more resources and information you may want to know about preparing and maintaining your site for search engines' robots and crawlers.

Meta Tags

Meta Tags contain information that describes your site's purpose, description, and keywords used within your site. The meta tags are stored within the head of your header.php template file. By default, they are not included in WordPress, but you can manually include them and the article on Meta Tags in WordPress takes you through the process of adding meta tags to your WordPress site.

The WordPress custom fields option can also be used to include keywords and descriptions for posts and Pages. There are also several WordPress Plugins that can also help you to add meta tags and keyword descriptions to your site found within the Codex's Plugins - Meta Tags list.

Feed Submissions

WordPress comes built-in with various feeds, allowing your site to be viewed by various feed readers. Many search engines are now accepting feed submissions, and there are many site which specialize in directories of feeds and feed services.

To submit your site's feeds, you need to know the link to the various feeds your site provides. The article WordPress Feeds lists the various links of the feeds that come built into WordPress.

For information on customizing these links, see the article on Customizing Feeds.

• Pingomatic(http://pingomatic.com/)
• Feed Shark(http://feedshark.brainbliss.com/)
• Robin Good's RSSTop55 - Best Blog Directory And RSS Submission Sites(http://www.masternewmedia.org/rss/top55/)
• Ari Paparo's Big List of Blog Search Engines and Feed Services(http://www.aripaparo.com/archive/000632.html)
• WordPress compressed all inclusive ping list(http://www.instant-info-online.com/wordpress-compressed-all-inclusive-ping-list.html)

Technorati Tags

Technorati(http://www.technorati.com/) is a "real-time search engine that keeps track of what is going on in the blogosphere ” the world of weblogs." According to the site, "Technorati tracks the number of links, and the perceived relevance of blogs, as well as the real-time nature of blogging. Because Technorati automatically receives notification from weblogs as soon as they are updated, it can track the thousands of updates per hour that occur in the blogosphere, and monitor the communities (who's linking to whom) underlying these conversations."

Technorati tags(http://www.technorati.com/tag/) are used to categorize the different topics and information used by blogs. Technorati uses WordPress categories as tags automatically. You can add more tags(http://www.technorati.com/help/tags.html) by adding a rel="tag" to any link on your site. For example:

<a href="http://wordpress.org/" rel="tag">WordPress</a>
 
<a href="http://codex.wordpress.org/" rel="tag">Codex</a>

There are also several WordPress plugins for maximizing Technorati tags on the Plugins - Technorati list.

Note: In WordPress v1.5.x, Technorati will automatically recognize your category names as tags. For more info, see: WordPress Categories, Technorati Tags and Search Engine Optimisation(http://www.tomrafteryit.net/wordpress-categories-technorati-tags-and-search-engine-optimisation/)

Note: For optimal Technorati listing, you should include the Atom feed in the header of your theme. For this you may use the following code, which you can add somewhere between the

<link rel="alternate" type="application/atom+xml"
title="Atom 0.3" href="<?php bloginfo('atom_url');?>" />

Without adding the Atom feed to your header, your posts in Technorati will most likely be displayed wrongly. Which can result in the post content on Technorati duplicating the post title and listing other (meta) information, which is displayed between the post title and the real post content on your weblog.

Permalinks

Permalinks are enhancements to your existing URLs which can improve search engine optimization by presenting your post, page, and archive URLs as something like http://example.com/2003/05/23/my-cheese-sandwich/ rather than http://example.com/index.php?p=423. See Using Permalinks for more information.

As search engines use links and the title as part of their information gathering, links to posts and articles within your site gain importance with Permalinks.

See also:

• Boost Search Engine Optimisation (SEO) using Permalinks(http://www.tomrafteryit.net/boost-search-engine-optimisation-seo-using-permalinks/)
• Permalinks Migration Plugin(http://www.deanlee.cn/wordpress/permalinks-migration-plugin/)With this plugin, you can safely change your permalink structure without breaking the old links to your website.

As an added bonus, enable the Permalink Redirect plugin(http://scott.yang.id.au/2005/05/permalink-redirect/). This plugin "replies a 301 permanent redirect, if request URI is different from entry's (or archive's) permalink. It is used to ensure that there is only one URL associated with each blog entry."

Sitemaps

A sitemap or "site map" is a single page listing of all the posts on your website. It is intended for your visitors to get a good overview on what your site is about and to find a blog post quickly but it also has great benefits in the search engines as a good link is always pointing to all your blog posts. By having a link to your sitemap on all your sites pages both visitors and search engines will easily get to it and find all your posts.

Here is a tutorial with three different examples of sitemaps with demos and how to set them up:

• Tutorial: Automatic Sitemap for WordPress(http://www.jimwestergren.com/tutorial-automatic-sitemap-in-wordpress/)

Google Sitemaps

As of June 2005, Google is now accepting sitemaps of your site as part of their website submissions. Google needs to have this sitemap formatted in a special way using XML. You can find more information about Google's Sitemap Submissions(https://www.google.com/webmasters/sitemaps/login) from Google, and the discussion on the WordPress Forum(http://www.wordpress.org/support) about WordPress and Google Site maps(http://wordpress.org/support/topic/35465).

Some utilities have been created to help the WordPress user to create a Google site map of their site for submission to Google. For more information on these and Google sitemaps:

• Inside Google Sitemaps (Official Google Blog)(http://sitemaps.blogspot.com/)
• Google Sitemap Generator Plugin for WordPress(http://www.arnebrachhold.de/2005/06/05/google-sitemaps-generator-v2-final)
• Make your sitemap human readable(http://blog.leion.net/2006/06/02/adding-human-readable-google-sitemap/)
• Simple Sitemap Generator(http://andreasjuch.mywol.org/mediawiki/Google_Sitemap_Generator_for_WordPress)
• Google Sitemaps - UltimateTagWarrior Tag Plugin Addon(http://www.dicontas.co.uk/blog/google-sitemap-utw-tag-wordpress-plugin/)

Link Relationships

There is some debate over whether listing the link relations actually effect search engine ranking however it is simple to implement.

• W3(http://www.w3.org/TR/REC-html40/struct/links.html#h-12.1.2)
• Linkrel Plugin(http://xmouse.ithium.net/essays/linkrels-another-wordpress-plugin/)
• rel Plugin(http://boxofchocolates.ca/archives/2004/06/12/link-relationship-plugin-for-wordpressLink)
• Theme Code(http://blog.noontide.ca/2006/03/29/wordpress-link-rel-next-prev-index/)

More Resources and Tutorials

There is a lot to learn about search engine optimization and site submission. Here are just a few sites to help you learn more about how this works:

• More WordPress Codex Articles on Website Development
• All About Search Indexing Robots and Spiders(http://www.searchtools.com/robots/)
• Boost Search Engine Optimization (SEO) Using Permalinks(http://www.tomrafteryit.net/boost-search-engine-optimisation-seo-using-permalinks/)
• Effectively Marketing Business Blogs(http://www.tomrafteryit.net/effectively-marketing-business-blogs/)
• Getting Ready for Search Engine Submissions(http://www.cameraontheroad.com/index.php?p=193)
• Preparation for Website Search Engine Submissions(http://www.cameraontheroad.com/index.php?p=168)
• Search Engine Optimization - Step 1. Identification of Aims(http://www.tomrafteryit.net/search-engine-optimisation-step-1-identification-of-aims/)
• Search Engine Optimization - Step 2. Benchmarking(http://www.tomrafteryit.net/search-engine-optimisation-step-2-benchmarking/)
• Search Engine Optimization - Step 3. Keyword and Keyphrase Research(http://www.tomrafteryit.net/search-engine-optimisation-step-3-keyword-and-keyphrase-research/)
• Search Engine Robots that Search Your Site (http://www.jafsoft.com/searchengines/webbots.html)
• Search Engine Test-Webpage Spider and Crawler Test(http://www.0email.net/)
• Search Engine Watch(http://www.searchenginewatch.com)
• Search Engine Watch Site Submission Tips and Tricks(http://searchenginewatch.com/webmasters/index.php)
• Search Engine Watch's Tips and Articles on Optimization(http://searchenginewatch.com/resources/article.php/2156511)
• How to Search the Web - Understanding How People Search(http://daphne.palomar.edu/TGSEARCH/)
• Understanding Search Results Pages(http://www.clickz.com/search/results/article.php/3291021)
• Website Development Step-by-Step(http://www.cameraontheroad.com/index.php?p=204)
• Why is Search Engine Optimization Important?(http://www.tomrafteryit.net/why-is-search-engine-optimisation-important/)
• WordPress Categories, Technorati Tags, and Search Engine Optimization(http://www.tomrafteryit.net/wordpress-categories-technorati-tags-and-search-engine-optimisation/)
• WordPress SEO: First Steps(http://seo.blorc.com/seo/wordpress-seo-first-steps/)
• SEO for WordPress Blogs (http://www.jimwestergren.com/seo-for-wordpress-blogs/)
• How to Optimize WordPress for Search Engines (complete guide)(http://www.geek-notes.com/wordpress/25/optimize-wordpress-for-search-engines/)
• WordPress search engine optimization without code modification(http://www.akamarketing.com/blog/30-wordpress-search-engine-optimization-without-code-modification.html)
• Using WordPress conditionals for search engine optimisation purposes(http://www.akamarketing.com/blog/31-using-wordpress-conditionals-for-search-engine-optimisation-purposes.html)
• Google to Yahoo MSN Sitemaps Creator Tool

SEO in WordPress originally appeared on AskApache.com

In 2003, Nick Kew released a new module that complements Apache's mod_proxy and is essential for reverse-proxying. Since then he gets regular questions and requests for help on proxying with Apache. In this article he attempts to give a comprehensive overview of the proxying and mod_proxy_html

This article was originally published at ApacheWeek in January 2004, and moved to ApacheTutor with minor updates in October 2006.

Web Proxies

A proxy server is a gateway for users to the Web at large. Users configure the proxy in their browser settings, and all HTTP requests are routed via the proxy. Proxies are typically operated by ISPs and network administrators, and serve several purposes: for example,

• to speed access to the Web by caching pages fetched, so that popular pages don't have to be re-fetched for every user who views them.
• to enable controlled access to the web for users behind a firewall.
• to filter or transform web content.

Reverse Proxies

A reverse proxy is a gateway for servers, and enables one web server to provide content from another transparently. As with a standard proxy, a reverse proxy may serve to improve performance of the web by caching; this is a simple way to mirror a website. But the most common reason to run a reverse proxy is to enable controlled access from the Web at large to servers behind a firewall.

The proxied server may be a webserver itself, or it may be an application server using a different protocol, or an application server with just rudimentary HTTP that needs to be shielded from the web at large. Since 2004, reverse proxying has been the preferred method of deploying JAVA/Tomcat applications on the Web, replacing the old mod_jk (itself a special-purpose reverse proxy module).

Proxying with Apache

The standard Apache module mod_proxy supports both types of proxy operation. Under Apache 1.x, mod_proxy only supported HTTP/1.0, but from Apache 2.0, it supports HTTP/1.1. This distinction is particularly important in a proxy, because one of the most significant changes between the two protocol versions is that HTTP/1.1 introduces rich new cache control mechanisms.

This article deals with running a reverse proxy with Apache 2. Users of earlier versions of Apache are encouraged to upgrade and take advantage of the altogether richer architecture and improved application support. At the time of writing, the reason most commonly cited for not upgrading is difficulties running PHP on Apache 2. I cannot speak from personal experience, but several well-informed sources tell me the difficulty lies with non-thread-safe code in PHP, and that it works well with Apache 2 if it is built with the non-threaded Prefork MPM.

The Apache Proxy Modules

So far, we have spoken loosely of mod_proxy. However, it's a little more complicated than that. In keeping with Apache's modular architecture, mod_proxy is itself modular, and a typical proxy server will need to enable several modules. Those relevant to proxying and this article include:

• mod_proxy: The core module deals with proxy infrastructure and configuration and managing a proxy request.
• mod_proxy_http: This handles fetching documents with HTTP and HTTPS.
• mod_proxy_ftp: This handles fetching documents with FTP.
• mod_proxy_connect: This handles the CONNECT method for secure (SSL) tunneling.
• mod_proxy_ajp: This handles the AJP protocol for Tomcat and similar backend servers.
• mod_proxy_balancer implements clustering and load-balancing over multiple backends.
• mod_cache, mod_disk_cache, mod_mem_cache: These deal with managing a document cache. To enable caching requires mod_cache and one or both of disk_cache and mem_cache.
• mod_proxy_html: This rewrites HTML links into a proxy's address space.
• mod_headers: This modifies HTTP request and response headers.
• mod_deflate: Negotiates compression with clients and backends.

Having mentioned the modules, I'm going to ignore caching for the remainder of this article. You may want to add it if you are concerned about the load on your network or origin servers, but the details are outside the scope of this article. I'm also going to ignore all non-HTTP protocols, and load balancing.

Building Apache for Proxying

With the exception of mod_proxy_html, the above are all included in the core Apache distribution. They can easily be enabled in the Apache build process. For example:

$ ./configure --enable-so --enable-mods-shared="proxy \
proxy_http proxy_ftp proxy_connect headers"
$ make
# make install

Of course, you may want other build options too, and you could just as well build the modules as static.

If you are adding proxying to an existing installation, you should use apxs instead:

# apxs -c -i [module-name].c
noting that mod_proxy itself is in two source files
(mod_proxy.c and proxy_util.c).

This leaves mod_proxy_html, which is not included in the core distribution. mod_proxy_html is a third-party module, and requires a third-party library libxml2. At the time of writing, libxml2 is installed as standard or packaged for most operating systems. If you don't have it, you can download it from xmlsoft.org and install it yourself. For the purposes of this article, we'll assume libxml2 is installed as /usr/lib/libxml2.so, with headers in /usr/include/libxml2/libxml/.

1. Check libxml2 is installed. If you have a version older than 2.5.10, then upgrade - there's a bug in earlier versions that can, in some particular cases, severely affect performance.
2. Download mod_proxy_html.c from http://apache.webthing.com/
3. Build mod_proxy_html with apxs:

# apxs -c -I/usr/include/libxml2 -i mod_proxy_html.c

A Reverse Proxy Scenario

Company example.com has a website at www.example.com, which has a public IP address and DNS entry, and can be accessed from anywhere on the Internet.

The company also has a couple of application servers which have private IP addresses and unregistered DNS entries, and are inside the firewall. The application servers are visible within the network - including the webserver, as "internal1.example.com" and "internal2.example.com", But because they have no public DNS entries, anyone looking at internal1.example.com from outside the company network will get a "no such host" error.

A decision is taken to enable Web access to the application servers. But they should not be exposed to the Internet directly, instead they should be integrated with the webserver, so that http://www.example.com/app1/any-path-here is mapped internally to http://internal1.example.com/any-path-here and http://www.example.com/app2/other-path-here is mapped internally to http://internal2.example.com/other-path-here. This is a typical reverse-proxy situation.

Configuring the Proxy

As with any modules, the first thing to do is to load them in httpd.conf (this is not necessary if we build them statically into Apache).

LoadModule  proxy_module         modules/mod_proxy.so
LoadModule  proxy_http_module    modules/mod_proxy_http.so
#LoadModule proxy_ftp_module     modules/mod_proxy_ftp.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule  headers_module       modules/mod_headers.so
LoadModule  deflate_module       modules/mod_deflate.so
LoadFile    /usr/lib/libxml2.so
LoadModule  proxy_html_module    modules/mod_proxy_html.so

For windows users this is slightly different: you'll need to load libxml2.dll rather than libxml2.so, and you'll probably need to load iconv.dll and xlib.dll as prerequisites to libxml2 (you can download them from zlatkovic.com, the same site that maintains windows binaries of libxml2). The LoadFile directive is the same.

Of course, you may not need all the modules. Two that are not required in our typical scenario are shown commented out above.

Having loaded the modules, we can now configure the Proxy. But before doing so, we have an important security warning:

Do Not set "ProxyRequests On". Setting ProxyRequests On turns your server into an Open Proxy. There are 'bots scanning the Web for open proxies. When they find you, they'll start using you to route around blocks and filters to access questionable or illegal material. At worst, they might be able to route email spam through your proxy. Your legitimate traffic will be swamped, and you'll find your server getting blocked by things like family filters.

Of course, you may also want to run a forward proxy with appropriate security measures, but that lies outside the scope of this article. The author runs both forward and reverse proxies on the same server (but under different Virtual Hosts).

The fundamental configuration directive to set up a reverse proxy is ProxyPass. We use it to set up proxy rules for each of the application servers:

ProxyPass       /app1/  http://internal1.example.com/
ProxyPass       /app2/  http://internal2.example.com/

Now as soon as Apache re-reads the configuration (the recommended way to do this is with "apachectl graceful"), proxy requests will work, so http://www.example.com/app1/some-path maps to http://internal1.example.com/some-path as required.

However, this is not the whole story. ProxyPass just sends traffic straight through. So when the application servers generate references to themselves (or to other internal addresses), they will be passed straight through to the outside world, where they won't work.

For example, an HTTP redirection often takes place when a user (or author) forgets a trailing slash in a URL. So the response to a request for http://www.example.com/app1/foo proxies to http://internal.example.com/foo which generates a response:

HTTP/1.1 302 Found
Location: http://internal.example.com/foo/
(etc)

But from the outside world, the net effect of this is a "No such host" error. The proxy needs to re-map the Location header to its own address space and return a valid URL

HTTP/1.1 302 Found
Location: http://www.example.com/app1/foo/

The command to enable such rewrites in the HTTP Headers is ProxyPassReverse. The Apache documentation suggests the form:

ProxyPassReverse /app1/ http://internal1.example.com/
ProxyPassReverse /app2/ http://internal2.example.com/

However, there is a slightly more complex alternative form that I recommend as more robust:

<location /app1/>
ProxyPassReverse /
</location>
<location /app2/>
ProxyPassReverse /
</location>

The reason for recommending this is that a problem arises with some application servers. Suppose for example we have a redirect:

HTTP/1.1 302 Found
Location: /some/path/to/file.html

This is a violation of the HTTP protocol and so should never happen: HTTP only permits full URLs in Location headers. However, it is also a source of much confusion, not least because the CGI spec has a similar Location header with different semantics where relative paths are allowed. There are a lot of broken servers out there! In this instance, the first form of ProxyPassReverse will return the incorrect response

HTTP/1.1 302 Found
Location: /some/path/to/file.html

which, even allowing for error-correcting browsers, is outside the Proxy's address space and won't work. The second form fixes this to

HTTP/1.1 302 Found
Location: /app2/some/path/to/file.html

which is still broken, but will at least work in error-correcting browsers. Most browsers will deal with this.

If your backend server uses cookies, you may also need the ProxyPassReverseCookiePath and ProxyPassReverseCookieDomain directives. These are similar to ProxyPassReverse, but deal with the different form of cookie headers. These require mod_proxy from Apache 2.2 (recommended), or a patched version of 2.0.

Fixing HTML Links

As we have seen, ProxyPassReverse remaps URLs in the HTTP headers to ensure they work from outside the company network. There is, however, a separate problem when links appear in HTML pages served. Consider the following cases:

1. This link will be resolved by the browser and will work correctly.
2. This link will be resolved by the browser to http://www.example.com/otherfile.html, which is incorrect.
3. This link will resolve to "no such host" for the browser.

The same problem of course applies to included content such as images, stylesheets, scripts or applets, and other contexts where URLs occur in HTML.

To fix this requires us to parse the HTML and rewrite the links. This is the purpose of mod_proxy_html. It works as an output filter, parsing the HTML and rewriting links as it is served. Two configuration directives are required to set it up:

• SetOutputFilter proxy-html This simply inserts the filter, to enable ProxyHTMLURLMap
• ProxyHTMLURLMap from-pattern to-pattern [flags] In its basic form, this has a similar purpose and semantics to ProxyPassReverse. Additionally, an extended form is available to enable search-and-replace rewriting of URLs within Scripts and Stylesheets.

How it works

mod_proxy_html is based on a SAX parser: specifically the HTMLparser module from libxml2 running in SAX mode (any other parse mode would of course be very much slower, especially for larger documents). It has full knowledge of all URI attributes that can occur in HTML 4 and XHTML 1. Whenever a URL is encountered, it is matched against applicable ProxyHTMLURLMap directives. If it starts with any from-pattern, that will be rewritten to the to-pattern. Rules are applied in the reverse order to their appearance in httpd.conf, and matching stops as soon as a match is found.

Here's how we set up a reverse proxy for HTML. Firstly, full links to the internal servers should be rewritten regardless of where they arise, so we have:

ProxyHTMLURLMap http://internal1.example.com /app1
ProxyHTMLURLMap http://internal2.example.com /app2

Note that in this instance we omitted the "trailing" slash. Since the matching logic is starts-with, we use the minimal matching pattern. We have now globally fixed case 3 above.

Case 2 above requires a little more care. Because the link doesn't include the hostname, the rewrite rule must be context-sensitive. As with ProxyPassReverse above, we deal with that using <location>

<location /app1/>
ProxyHTMLURLMap / /app1/
</location>
<location /app2/>
ProxyHTMLURLMap / /app2/
</location>

Debugging your Proxy Configuration

The above is a simple case taken from mod_proxy_html version 1. With the more complex URLmapping and rewriting enabled by Version 2, you may need a bit of help setting up a complex ruleset, perhaps involving a series of complex regexps, chained anc blocking rules, etc. To help with setting up and troubleshooting your rulesets, mod_proxy_html 2 provides a "debug" mode, in which all the 'interesting' things it does are written to the Apache error log. To analyse and fix your rulesets, set

ProxyHTMLLogVerbose On
LogLevel Info   (or LogLevel Debug)

Now run your testcases through your rulesets, and examine the apache error log for details of exactly how it was processed.

Do not leave ProxyHTMLLogVerbose On for normal use. Although the effect is marginal, it is an overhead.

Extended URL Mapping

The previous section sets up remapping of HTML URLs, but leaves any URL encountered in a Stylesheet or Script untouched. mod_proxy_html doesn't parse Javascript or CSS, so dealing with URLs in them requires text-based search-and-replace. This is enabled by the directive ProxyHTMLExtended On.

Because the extended mode is text-based, it can no longer guarantee to match exact URLs. It's up to you to devise matching rules that can pick out URLs, just as if you were writing an old-fashioned Perl or PHP regexp-based filter (though of course it's still massively more efficient than performing search-and-replace on an entire document in-memory). To help with this, ProxyHTMLExtended supports both simple text-based and regular expression search-and-replace, according to the flags. You can also use the flags to specify rules separately for HTML links, scripting events, and embedded scripts and stylesheets.

A second key consideration with extended URL mapping is that whereas an HTML link contains exactly one URL, a script or stylesheet may contain many. So instead of stopping after a successful match, the processor will apply all applicable mapping rules. This can be stopped with the L (last) flag.

Dealing with multimedia content

We just set up a proxy to parse and where necessary correct HTML. But of course, the web isn't just HTML. Surely feeding non-HTML content through an HTML parser is at best inefficient, if not totally broken?

Yes indeed. mod_proxy_html deals with that by checking the Content-Type header, and removing itself from the processing chain when a document is not HTML (text/html) or XHTML (application/xhtml+xml). This happens in the filter initialisation phase, before any data are processed by the filter.

But that still leaves a problem. Consider compressed HTML:

Content-Type: text/html
Content-Encoding: gzip

Feeding that into an HTML parser is clearly broken!

There are two solutions to this. One is to uncompress the incoming data with mod_deflate. Uncompressing and compressing content radically reduces network traffic, but increases the processor load on the proxy. It is worthwhile if and only if bandwidth between the proxy and the backend is at a premium: this is common on the 'net at large, but unlikely to be the case on a company internal network.

SetOutputFilter  INFLATE;proxy-html;DEFLATE

The alternative solution is to refuse to support compression. Stripping any Accept-Encoding request header does the job. So invoking mod_headers, we add a directive

RequestHeader unset Accept-Encoding

This should only apply to the Proxy, so we put it inside our <location> containers.

A similar situation arises in the case of encrypted (https) content. But in this case, there is no such workaround: if we could decrypt the data to process it then so could any other man-in-the-middle, and the security would be worthless. This can only be circumvented by installing mod_ssl and a certificate on the proxy, so that the actual secure session is between the browser and the proxy, not the origin server.

The Complete Configuration

We are now in a position to write a complete configuration for our reverse proxy. Here is a bare minimum, that ignores extended urlmapping:

LoadModule proxy_module      modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule headers_module    modules/mod_headers.so
LoadFile   /usr/lib/libxml2.so
LoadModule proxy_html_module modules/mod_proxy_html.so
 
ProxyRequests off
ProxyPass /app1/ http://internal1.example.com/
ProxyPass /app2/ http://internal2.example.com/
ProxyHTMLURLMap http://internal1.example.com /app1
ProxyHTMLURLMap http://internal2.example.com /app2
 
<location /app1/>
ProxyPassReverse /
SetOutputFilter  proxy-html
ProxyHTMLURLMap  /      /app1/
ProxyHTMLURLMap  /app1  /app1
RequestHeader    unset  Accept-Encoding
</location>
 
<location /app2/>
ProxyPassReverse /
SetOutputFilter proxy-html
ProxyHTMLURLMap /       /app2/
ProxyHTMLURLMap /app2   /app2
RequestHeader   unset   Accept-Encoding
</location>

Of course, there's more than one way to do it. Our configuration would actually have been simpler if we'd used Virtual Hosts for each application server. But that takes you beyond the realm of Apache configuration and into DNS. If you don't fully understand that (or if you think "why can't I see my domain" is a webserver question), then please don't try using virtual hosts for this.

Further topics

Caching

We haven't dealt with caching in this article. In a company-intranet situation, the connection from the proxy to the application servers is the local LAN, which is probably fast and has ample capacity. In such cases, caching at the proxy will have little effect, and can probably be omitted.

If we want to cache pages, we can of course do so with mod_cache But that is beyond the scope of this article.

Content Transformation

Another powerful use for a proxy is to transform the content on-the-fly according to the user's preferences. This author's flagship mod_accessibility product (from which mod_proxy_html is a spinoff) serves to transform HTML and XHTML on-demand to enhance usability and accessibility.

Filtering and Security

A reverse proxy is not the natural place for a "family filter", but is ideal for defining access controls and imposing security restrictions. We could, for example, configure the proxy to recognise a custom header from an origin server and block content based on it. This delegates control to the application servers.

Questions and Answers

(Q) Where can I get the software?

(A) Most of it from the obvious place, http://httpd.apache.org/ mod_proxy_html is available from http://apache.webthing.com/ libxml2 is available from http://xmlsoft.org/. Windows users should read libxml2.dll for libxml2.so, and can obtain it together with the prerequisites iconv.dll and zlib.dll from Igor Zlatkovic's site.

(Q) Can I get a binaries of software ?

(A) If there's no link at the websites above, ask the provider of your operating system or distribution. The author can compile it on different platforms but does not provide a free compilation service.

(Q) What is httpd.conf? My apache has different configuration files.

(A) Some distribution packagers mess about with the Apache configuration. If this applies to you, the details should be documented by your distributor, and have nothing to do with Apache itself! Substitute your distributions choice of configuration file for httpd.conf in the above discussion, or create your own proxy.conf file and Include it.

(Q) You mentioned apxs and apachectl. Where do I find them?

(A) They're part of a standard Apache installation (except on Windows). If you don't have them or can't find them, that's a problem with your installation. The easiest solution is probably to download a complete Apache from httpd.apache.org.

(Q) Does mod_proxy_html deal with Javascript links?

(A) From mod_proxy_html 2.0, yes!

(Q) The proxy appears to change my HTML?

(A) It doesn't really, but it may appear to. Here are the possible causes:

1. Changing the FPI (the <!DOCTYPE ...> line) may affect some browsers. FIX: set the doctype explicitly if this bothers you.

2. mod_proxy_html has the side-effect of transforming content to utf-8 (Unicode) encoding. This should not be a problem: utf-8 is well-supported by browsers, and offers comprehensive support for internationalisation. If it appears to cause a problem, that's almost certainly a bug in the application server, or possibly a misconfigured browser. FIX: filter through mod_charset_lite to your chosen charset.

3. mod_proxy_html will perform some minor normalisations. If your HTML includes elements that are closed implicitly, it will explicitly close them. In other words:

   <body>
   <p>Hello, World!
   </body>

   will be transformed to

   <body>
   <p>Hello, World!</p>
   <body>

   If this affects the rendition in your browser, it almost certainly means you are using malformed HTML and relying on error-correction in a browser. FIX: validate your HTML! The online Page Valet service will both validate and show your markup normalised by the DTD, while a companion tool AccessValet will show markup normalised by the same parser used in the proxy, and highlight other problems. Both are available at http://valet.webthing.com/

(Q) I need a customised solution.

(A) The author is available for development and consultancy.

Running a Reverse Proxy in Apache originally appeared on AskApache.com