This is part 2 of the exhaustive Htaccess Tutorial. I realized it was so lengthy that search engines and visitors were having real problems with it, so I moved half of it here. But this gave me the opportunity to add a ton of new stuff that I hadn't been able to add to the main htaccess tutorial. And now this new part 2 is already twice as big as the original! To sum up, this is a work in progress.

Moving Half of Original Tutorial to a new URL - SEO Rewriting

Well, the original tutorial was at /htaccess/apache-htaccess.html, but it was such a huge article that search engines were dropping it! So I split it into 3 new urls.

1. /htaccess/htaccess.html - 1
2. /htaccess/htaccess-rewrites.html - Rewrites
3. /htaccess/apache-htaccess-2.html - Part 2

The /htaccess/apache-htaccess-2.html is a lot of unfinished new stuff, and the /htaccess/htaccess-rewrites.html (this very page) still needs to be split several times to get the filesize down.

301 Link Juice

The /apache-htaccess.html url has been around since 2006, and it was full of original ideas and examples for using htaccess. Almost every htaccess guide or tutorial published since then has many of the examples and ideas from that tutorial, (I encourage people to modify and republish everything on this site according to copyright). I know it, the authors sometimes know it, but the main point is Google for sure knows it. That's the beauty of creating unique content, Google sees that. So that /apache-htaccess.html link has extreme link juice, from all the sites, books, papers, and presentations that have linked to it since 2006.

Filesize Is important

I like to use my blog as a way to keep notes about my research organized, and I am extremely good at doing research, unfortunately, that means I have a huge article. The filesize for the html alone is larger than all the other resources like javascript and images, combined.

• So that means it is very difficult to view on a mobile device, or a slow connection.
• For google-bot and other search engine crawlers and robots, this is a huge problem (I made a big mistake letting it get that size).
• The robot has to parse an enormous single html file, containing hundreds/thousands of external and internal links, and its such an issue they can decide to just skip indexing that url until the filesize is manageable.
• Once again, it's all about the human experience, a huge single file is not good for anyone who isn't printing it out to read offline.

Starting Fresh without losing juice

So since the page wasn't being indexed much since it was so huge, I decided to split up the content into new separate urls and utilize a 301 Redirect to transfer all the link juice from /apache-htaccess.html to /htaccess.html.

301 Redirect Timeline

1. Now, the idea is to take the first pages from the original multi-page guide, do a little improvement on that content, and save it to the new url /htaccess.html.
2. Then just continue taking the next page from the remaining original guide and creating new pages using the original links structure. /apache-htaccess-2.html, /apache-htaccess-3.html, etc.. This is a secondary backup to the new /htaccess.html url, which will receive the 301 link juice from the old url, but these secondary pages will help keep the links on external sites good.
3. Then, I setup a 301 Redirect in my .htaccess file to redirect the old url to the new /htaccess.html url.
4. Finally, I delete the old url and it is replaced forever by a 301 Redirect pointing to my new location, filesize problems eliminated.

301 Redirect Code Used

This is so easy to do with RedirectMatch, way faster and easier than using mod_rewrite to handle this, and much less overhead. Note that this is a general command that I will leave up for a few weeks and slowly tighten it up by looking at my Google Analytics and Apache Logs. For instance, this first redirectmatch rule is an older RedirectMatch I still have active to redirect all the old links pointing at /2006/htaccess/apache-htaccess.html to /htaccess/apache-htaccess.html from when I ditched the date-based permalinks back in 2007, and you can see it is a little tighter than the 2nd one which also redirects requests for apache-htaccess.html/feed/ or trackback or whatever.

RedirectMatch 301 ^/2006/.*apache-htaccess.html$ http://www.askapache.com/htaccess/apache-htaccess.html
RedirectMatch 301 ^/.*apache-htaccess.html(.*)$ http://www.askapache.com/htaccess/htaccess.html$1

Advanced 301 Redirects for SEO

I wrote a couple of articles that go into detail about maximizing the SEO with linking and redirects, it remains one of my most helpful articles to anyone trying to rank higher the right way, the Google way.. SEO Secrets of AskApache Part 2

Htaccess Rewrites for Moving Urls originally appeared on AskApache.com

Htaccess Rewrites are enabled by using the Apache module mod_rewrite, which is one of the most powerful Apache modules and features availale. Htaccess Rewrites through mod_rewrite provide the special ability to Rewrite requests internally as well as Redirect request externally.

When the url in your browser's location bar stays the same for a request it is an internal rewrite, when the url changes an external redirection is taking place. This is one of the first, and one of the biggest mental-blocks people have when learning about mod_rewrite... But I have a secret weapon for you to use, a new discovery from years of research that makes learning mod_rewrite drastically quicker and easier. It truly does or I wouldn't be saying so in the introduction of this article.

Despite the tons of examples and docs, mod_rewrite is voodoo. Damned cool voodoo, but still voodoo.
-- Brian Moore

Note: After years of fighting to learn my way through rewriting urls with mod_rewrite, I finally had a breakthrough and found a way to outsmart the difficulty of mod_rewrite that I just couldn't seem to master. The Mod_Rewrite RewriteCond/RewriteRule Variable Value Cheatsheet is the one-of-a-kind tool that changed the game for me and made mod_rewriting no-harder than anything else.

So keep that mod_rewrite reference bookmarked and you will be able to figure out any RewriteRule or RewriteCond, an amazing feat considering it took me a LONG time to figure this stuff out on my own. But that was before the craziness, one of the most challenging and productive .htaccess experiments I've done... An experiment so ILL it's sick like a diamond disease on your wrist! $$$. That mod_rewrite experiment/tutorial was the culmination of many different advanced mod_rewrite experiments I had done in the past and included most of my very best .htaccess tricks. With the cheatsheet it's no longer Voodoo.. Its just what you do. Now lets dig in!

Htaccess rewrites TOC

• .htaccess rewrite examples should begin with:
• Require the www
• Require no www
• Check for a key in QUERY_STRING
• Removes the QUERY_STRING from the URL
• Fix for infinite loops
• Redirect .php files to .html files (SEO friendly)
• Redirect .html files to actual .php files (SEO friendly)
• block access to files during certain hours of the day
• Rewrite underscores to hyphens for SEO URL
• Require the www without hardcoding
• Require no subdomain
• Require no subdomain
• Redirecting WordPress Feeds to Feedburner
• Only allow GET and PUT request methods
• Prevent Files image/file hotlinking and bandwidth stealing
• Stop browser prefetching

If you really want to take a look, check out the mod_rewrite.c and mod_rewrite.h files.

Be aware that mod_rewrite (RewriteRule, RewriteBase, and RewriteCond) code is executed for each and every HTTP request that accesses a file in or below the directory where the code resides, so it's always good to limit the code to certain circumstances if readily identifiable.

For example, to limit the next 5 RewriteRules to only be applied to .html and .php files, you can use the following code, which tests if the url does not end in .html or .php and if it doesn't, it will skip the next 5 RewriteRules.

RewriteRule !\.(html|php)$ - [S=5]
RewriteRule ^.*-(vf12|vf13|vf5|vf35|vf1|vf10|vf33|vf8).+$ - [S=1]

.htaccess rewrite examples should begin with:

Options +FollowSymLinks
 
RewriteEngine On
RewriteBase /

Require the www

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Loop Stopping Code

Sometimes your rewrites cause infinite loops, stop it with one of these rewrite code snippets.

RewriteCond %{REQUEST_URI} ^/(stats/|missing\.html|failed_auth\.html|error/).* [NC]
RewriteRule .* - [L]
 
RewriteCond %{ENV:REDIRECT_STATUS} 200
RewriteRule .* - [L]

Cache-Friendly File Names

This is probably my favorite, and I use it on every site I work on. It allows me to update my javascript and css files in my visitors cache's simply by naming them differently in the html, on the server they stay the same name. This rewrites all files for /zap/j/anything-anynumber.js to /zap/j/anything.js and /zap/c/anything-anynumber.css to /zap/c/anything.css

RewriteRule ^zap/(j|c)/([a-z]+)-([0-9]+)\.(js|css)$ /zap/$1/$2.$4 [L]

SEO friendly link for non-flash browsers

When you use flash on your site and you properly supply a link to download flash that shows up for non-flash aware browsers, it is nice to use a shortcut to keep your code clean and your external links to a minimum. This code allows me to link to site.com/getflash/ for non-flash aware browsers.

RewriteRule ^getflash/?$ http://www.adobe.com/shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash [NC,L,R=307]

Removing the Query_String

On many sites, the page will be displayed for both page.html and page.html?anything=anything, which hurts your SEO with duplicate content. An easy way to fix this issue is to redirect external requests containing a query string to the same uri without the query_string.

RewriteCond %{THE_REQUEST} ^GET\ /.*\;.*\ HTTP/
RewriteCond %{QUERY_STRING} !^$
RewriteRule .* http://www.askapache.com%{REQUEST_URI}? [R=301,L]

Sending requests to a php script

This .htaccess rewrite example invisibly rewrites requests for all Adobe pdf files to be handled by /cgi-bin/pdf-script.php

RewriteRule ^(.+)\.pdf$  /cgi-bin/pdf-script.php?file=$1.pdf [L,NC,QSA]

Setting the language variable based on Client

For sites using multiviews or with multiple language capabilities, it is nice to be able to send the correct language automatically based on the clients preferred language.

RewriteCond %{HTTP:Accept-Language} ^.*(de|es|fr|it|ja|ru|en).*$ [NC]
RewriteRule ^(.*)$ - [env=prefer-language:%1]

Deny Access To Everyone Except PHP fopen

This allows access to all files by php fopen, but denies anyone else.

RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^.+$ [NC]
RewriteRule .* - [F,L]

If you are looking for ways to block or deny specific requests/visitors, then you should definately read Blacklist with mod_rewrite. I give it a 10/10

Deny access to anything in a subfolder except php fopen

This can be very handy if you want to serve media files or special downloads but only through a php proxy script.

RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+)/.*\ HTTP [NC]
RewriteRule .* - [F,L]

Require no www

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^askapache\.com$ [NC]
RewriteRule ^(.*)$ http://askapache.com/$1 [R=301,L]

Check for a key in QUERY_STRING

Uses a RewriteCond Directive to check QUERY_STRING for passkey, if it doesn't find it it redirects all requests for anything in the /logged-in/ directory to the /login.php script.

RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} !passkey
RewriteRule ^/logged-in/(.*)$ /login.php [L]

Removes the QUERY_STRING from the URL

If the QUERY_STRING has any value at all besides blank than the?at the end of /login.php? tells mod_rewrite to remove the QUERY_STRING from login.php and redirect.

RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} .
RewriteRule ^login.php /login.php? [L]

Fix for infinite loops

An error message related to this isRequest exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.or you may seeRequest exceeded the limit,probable configuration error,Use 'LogLevel debug' to get a backtrace, orUse 'LimitInternalRecursion' to increase the limit if necessary

RewriteCond %{ENV:REDIRECT_STATUS} 200
RewriteRule .* - [L]

External Redirect .php files to .html files (SEO friendly)

RewriteRule ^(.*)\.php$ /$1.html [R=301,L]

Internal Redirect .php files to .html files (SEO friendly)

Redirects all files that end in .html to be served from filename.php so it looks like all your pages are .html but really they are .php

RewriteRule ^(.*)\.html$ $1.php [R=301,L]

block access to files during certain hours of the day

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
# If the hour is 16 (4 PM) Then deny all access
RewriteCond %{TIME_HOUR} ^16$
RewriteRule ^.*$ - [F,L]

Rewrite underscores to hyphens for SEO URL

Converts all underscores "_" in urls to hyphens "-" for SEO benefits... See the full article for more info.

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
 
RewriteRule !\.(html|php)$ - [S=4]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_([^_]*)_(.*)$ $1-$2-$3-$4-$5 [E=uscor:Yes]
RewriteRule ^([^_]*)_([^_]*)_([^_]*)_(.*)$ $1-$2-$3-$4 [E=uscor:Yes]
RewriteRule ^([^_]*)_([^_]*)_(.*)$ $1-$2-$3 [E=uscor:Yes]
RewriteRule ^([^_]*)_(.*)$ $1-$2 [E=uscor:Yes]
 
RewriteCond %{ENV:uscor} ^Yes$
RewriteRule (.*) http://d.com/$1 [R=301,L]

Require the www without hardcoding

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.[a-z-]+\.[a-z]{2,6} [NC]
RewriteCond %{HTTP_HOST} ([a-z-]+\.[a-z]{2,6})$     [NC]
RewriteRule ^/(.*)$ http://%1/$1 [R=301,L]

Require no subdomain

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} \.([a-z-]+\.[a-z]{2,6})$ [NC]
RewriteRule ^/(.*)$ http://%1/$1 [R=301,L]

Require no subdomain

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} \.([^\.]+\.[^\.0-9]+)$
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

Redirecting WordPress Feeds to Feedburner

Full article:Redirecting WordPress Feeds to Feedburner

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} ^/feed\.gif$
RewriteRule .* - [L]
 
RewriteCond %{HTTP_USER_AGENT} !^.*(FeedBurner|FeedValidator) [NC]
RewriteRule ^feed/?.*$ http://feeds.feedburner.com/apache/htaccess [L,R=302]
 
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

Only allow GET and PUT Request Methods

Article: Request Methods

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_METHOD} !^(GET|PUT)
RewriteRule .* - [F]

Prevent Files image/file hotlinking and bandwidth stealing

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?askapache.com/.*$ [NC]
RewriteRule \.(gif|jpg|swf|flv|png)$ /feed/ [R=302,L]

Stop browser prefetching

RewriteEngine On
SetEnvIfNoCase X-Forwarded-For .+ proxy=yes
SetEnvIfNoCase X-moz prefetch no_access=yes
 
# block pre-fetch requests with X-moz headers
RewriteCond %{ENV:no_access} yes
RewriteRule .* - [F,L]

This module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly. It supports an unlimited number of rules and an unlimited number of attached rule conditions for each rule, to provide a really flexible and powerful URL manipulation mechanism. The URL manipulations can depend on various tests, of server variables, environment variables, HTTP headers, or time stamps. Even external database lookups in various formats can be used to achieve highly granular URL matching.

This module operates on the full URLs (including the path-info part) both in per-server context (httpd.conf) and per-directory context (.htaccess) and can generate query-string parts on result. The rewritten result can lead to internal sub-processing, external request redirection or even to an internal proxy throughput.

Further details, discussion, and examples, are provided in the detailed mod_rewrite documentation.

Directives

• RewriteBase
• RewriteCond
• RewriteEngine
• RewriteLock
• RewriteLog
• RewriteLogLevel
• RewriteMap
• RewriteOptions
• RewriteRule

If you aren't already comfortable using mod_rewrite then I recommend this excellent mod_rewrite guide by one of my favorite mod_rewrite gurus that I've met.

htaccess Guide Sections

• htaccess tricks for Webmasters
• HTTP Header control with htaccess
• PHP on Apache tips and tricks
• SEO Redirects without mod_rewrite
• mod_rewrite examples, tips, and tricks
• HTTP Caching and Site Speedups
• Authentication on Apache
• htaccess Security Tricks and Tips
• SSL tips and examples
• Variable Fun (mod_env) Section
• .htaccess Security with MOD_SECURITY
• SetEnvIf and SetEnvIfNoCase Examples

« Search Engine Friendly Redirects | .htaccess Tutorial Index | » Speed up your site with Caching and cache-control

Htaccess Rewrites – Rewrite Tricks and Tips originally appeared on AskApache.com

We've figured out what mod_rewrite variables look like so we can create rewriterules and condition patterns based on the actual value. This cheatsheet is where we'll lay them all out for quick reference. This cheatsheet changed my life, way more than 301 redirect htaccess.

mod_rewrite Variable Value Cheat-Sheet

Jump to: API_VERSION, AUTH_TYPE, CONTENT_LENGTH, CONTENT_TYPE, DOCUMENT_ROOT, GATEWAY_INTERFACE, HTTPS, HTTP_ACCEPT, HTTP_ACCEPT_CHARSET, HTTP_ACCEPT_ENCODING, HTTP_ACCEPT_LANGUAGE, HTTP_CACHE_CONTROL, HTTP_CONNECTION, HTTP_COOKIE, HTTP_FORWARDED, HTTP_HOST, HTTP_KEEP_ALIVE, HTTP_PROXY_CONNECTION, HTTP_REFERER, HTTP_USER_AGENT, IS_SUBREQ, ORIG_PATH_INFO, ORIG_PATH_TRANSLATED, ORIG_SCRIPT_FILENAME, ORIG_SCRIPT_NAME, PATH, PATH_INFO, PHP_SELF, QUERY_STRING, REDIRECT_QUERY_STRING, REDIRECT_REMOTE_USER, REDIRECT_STATUS, REDIRECT_URL, REMOTE_ADDR, REMOTE_HOST, REMOTE_IDENT, REMOTE_PORT, REMOTE_USER, REQUEST_FILENAME, REQUEST_METHOD, REQUEST_TIME, REQUEST_URI, SCRIPT_FILENAME, SCRIPT_GROUP, SCRIPT_NAME, SCRIPT_URI, SCRIPT_URL, SCRIPT_USER, SERVER_ADDR, SERVER_ADMIN, SERVER_NAME, SERVER_PORT, SERVER_PROTOCOL, SERVER_SIGNATURE, SERVER_SOFTWARE, THE_REQUEST, TIME, TIME_DAY, TIME_HOUR, TIME_MIN, TIME_MON, TIME_SEC, TIME_WDAY, TIME_YEAR, TZ, UNIQUE_ID

API_VERSION: 20020903:12

RewriteCond %{API_VERSION} ^(.*)$
RewriteRule .* http://www.askapache.com?API_VERSION=%1 [R=307,L]

AUTH_TYPE: Digest

RewriteRule .* - [E=IN_AUTH_TYPE:%{AUTH_TYPE}]
RequestHeader set AUTH_TYPE "%{IN_AUTH_TYPE}e"

CACHE_CONTROL: max-age=0

RewriteCond %{ENV:CACHE_CONTROL} no-cache [NC]
RewriteRule . %{REQUEST_URI}?nocache [L]

CONNECTION: keep-alive

CONTENT_LENGTH: (null)

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP:Content-Length}%{CONTENT_LENGTH} ^$
RewriteRule .* - [F,NS,L]

CONTENT_TYPE: (null)

DOCUMENT_ROOT: /home/webroot/askapache.com

RewriteCond %{DOCUMENT_ROOT}/cache%{REQUEST_URI}/index.html -f
RewriteRule . /cache%{REQUEST_URI}/index.html

HOST: www.askapache.com

RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule . http://www.askapache.com%{REQUEST_URI} [R=301,L]

HTTP:

RewriteCond %{HTTP:Accept-Encoding} gzip [NC]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}.gz -f
RewriteRule . %{REQUEST_URI}.gz [L]

HTTPS: off

RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

HTTP_COOKIE: __qca=1176541205adf28-5553185; ASKAPACHEID=fdadfa4f33e62a878468; __utmc=1df3893

RewriteCond %{HTTP_COOKIE} ^.*autostart=on.*$
RewriteRule ^(.*)\.swf$ /$1\?autostart=true [NE,L]

HTTP_HOST: www.askapache.com

HTTP_REFERER: http://www.askapache.com/pro/mod_rewrite/catch.php?k=i

RewriteCond %{HTTP_REFERER} badhost [NC]
RewriteRule . - [F]

HTTP_USER_AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Firefox/3.0.1

RewriteCond %{HTTP_USER_AGENT} ^.*(Android|2.0\ MMP|240x320|AvantGo|BlackBerry|Blazer|Cellphone|Danger|DoCoMo|Elaine/3.0|EudoraWeb|hiptop|IEMobile|iPhone).*$ [NC]
RewriteRule ^(.+)$ /mobile/$1 [L]

IS_SUBREQ: false

KEEP_ALIVE: 300

PATH: /bin:/usr/bin:/sbin:/usr/sbin

QUERY_STRING: k=i

RewriteCond %{QUERY_STRING} showtime [NC]
RewriteCond T:%{TIME}_TY:%{TIME_YEAR}_TMO:%{TIME_MON}_TWD:%{TIME_WDAY}_TD:%{TIME_DAY}_TH:%{TIME_HOUR}_TMI:%{TIME_MIN}_TS:%{TIME_SEC} ^(.*)$
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI}?time=%1 [R,L]

REMOTE_ADDR: 22.162.134.211

RewriteCond %{REMOTE_ADDR} !^22\.162\.134\.211$
RewriteRule . http://www.askapache.com/maintenance-in-progress.html [R=307,L]

REMOTE_HOST: 22.162.134.211

REMOTE_PORT: 4220

REMOTE_USER: askapache

RewriteRule .* - [E=REMOTE_USER:%{HTTP:Authorization}]

REQUEST_FILENAME: /home/webroot/askapache.com/pro/mod_rewrite/index.php

REQUEST_METHOD: GET

RewriteCond %{REQUEST_METHOD} !^(POST|GET|HEAD|PROPFIND|OPTIONS)$
RewriteRule .* - [F,L]

REQUEST_PROTOCOL: HTTP/1.1

RewriteCond %{REQUEST_PROTOCOL} !^HTTP/(0\.9|1\.[01])$
RewriteRule . [F,L]

REQUEST_URI: /pro/mod_rewrite/index.php

RewriteCond %{REQUEST_URI} ^(robots\.txt|favicon\|ico)$ [NC]
RewriteRule . - [S=1]
RewriteCond %{HTTP_HOST} ^www
RewriteRule .* http://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

SCRIPT_FILENAME: /home/webroot/askapache.com/pro/mod_rewrite/index.php

SCRIPT_GROUP: daemong

SCRIPT_URI: http://www.askapache.com/pro/mod_rewrite/index.php

SCRIPT_URL: /pro/mod_rewrite/index.php

SCRIPT_USER: askapache

SERVER_ADDR: 208.113.134.190

SERVER_ADMIN: webmaster@askapache.com

SERVER_NAME: www.askapache.com

SERVER_PORT: 80

SERVER_PROTOCOL: HTTP/1.1

SERVER_SOFTWARE: Apache/2.0.61 (Unix) PHP/5.5 OpenSSL/0.9.7e

THE_REQUEST: GET /pro/mod_rewrite/index.php?k=i HTTP/1.1

RewriteCond %{THE_REQUEST} ^(GET|POST)\ /.*\?(s|search)=(.+)\ HTTP/ [NC]
RewriteRule .* http://www.askapache.com/search/%3/? [R=302,L,NE]

TIME: 20080915152142

RewriteCond %{QUERY_STRING} showtime [NC]
RewriteCond T:%{TIME}_TY:%{TIME_YEAR}_TMO:%{TIME_MON}_TWD:%{TIME_WDAY}_TD:%{TIME_DAY}_TH:%{TIME_HOUR}_TMI:%{TIME_MIN}_TS:%{TIME_SEC} ^(.*)$
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI}?time=%1 [R,L]

TIME_DAY: 15

TIME_HOUR: 15

TIME_MIN: 21

TIME_MON: 09

TIME_SEC: 42

TIME_WDAY: 1

TIME_YEAR: 2008

UNIQUE_ID: qOr5tEBvcm8AAE-VoiUAAAAQ

This article is meant to prepare us for the advanced mod_rewrite examples that are soon to be published. The upcoming article is going to be examples using mod_rewrite to achieve some crazy stuff... Here the focus is on identifying mod_rewrite variables and defining the limits of the module by checking the mod_rewrite source code.

For a broader mod_rewrite cheat sheet, check this printable cheat sheet.

Directives in Mod_Rewrite

RewriteEngine

RewriteEngine on|off

On or Off to enable or disable (default) the whole rewriting engine

RewriteOptions

RewriteOptions Options

List of option strings to set

RewriteBase

RewriteBase URL-path

the base URL of the per-directory context

RewriteCond

RewriteCond TestString CondPattern

an input string and a to be applied regexp-pattern

RewriteRule

RewriteRule Pattern Substitution [flags]

an URL-applied regexp-pattern and a substitution URL

RewriteMap

RewriteMap MapName MapType:MapSource

a mapname and a filename

RewriteLock

RewriteLock file-path

the filename of a lockfile used for inter-process synchronization

RewriteLog

RewriteLog file-path

the filename of the rewriting logfile

RewriteLogLevel

RewriteLogLevel Level

the level of the rewriting logfile verbosity (0=none, 1=std, .., 9=max)

RewriteRule Flags

C

Using the [Chain], or [C] flag, allows you to indicate that several rules should be chained together as a single logical transation. This is usually used when a transformation is sufficiently complicated to warrant breaking into several smaller steps.

E

'env|E=VAR:VAL' (set environment variable)

RewriteRule (root|cmd)\.exe - [E=worm:nimda]

F

'forbidden|F' (force URL to be forbidden)

G

'gone|G' (force URL to be gone)

H

'handler|H=Content-handler' (force Content handler)

L

'last|L' (last rule)

N

'next|N' (next round)

NC

'nocase|NC' (no case)

NE

'noescape|NE' (no URI escaping of output)

NS

'nosubreq|NS' (not for internal sub-requests)

P

'proxy|P' (force proxy)

PT

'passthrough|PT' (pass through to next handler)

QSA

'qsappend|QSA' (query string append)

R

'redirect|R  [=code]' (force redirect)

S

'skip|S=num' (skip next rule(s))

This flag forces the rewriting engine to skip the next num rules in sequence, if the current rule matches. Use this to make pseudo if-then-else constructs: The last rule of the then-clause becomes skip=N, where N is the number of rules in the else-clause. (This is not the same as the 'chain|C' flag!)

T

'type|T=MIME-type' (force MIME type)

Force the MIME-type of the target file to be MIME-type. This can be used to set up the content-type based on some conditions. For example, the following snippet allows .php files to be displayed by mod_php if they are called with the .phps extension:

Rules and Conditions Processing Order

1. The Pattern of the RewriteRule (^/.*$) is checked first.
2. If the pattern matches, then the RewriteCond's are checked.
3. If the RewriteConditions also match, the RewriteRule is applied.

RewriteRule Special Variables

1. ENV_
2. SSL_
3. HTTP_
4. LA-U_
5. LA-F_

RewriteCond Tests

• f - FILE_EXISTS
• s - FILE_SIZE
• l - FILE_LINK
• d - FILE_DIR
• x - FILE_XBIT
• U - LU_URL
• F - LU_FILE
• > - STR_GT
• < - STR_LT
• = - STR_EQ

Special Rewrite Redirects

1. "permanent" - HTTP_MOVED_PERMANENTLY
2. "temp" - HTTP_MOVED_TEMPORARILY
3. "seeother" - HTTP_SEE_OTHER
4. digit

Recognized by Mod_Rewrite

1. ajp://
2. balancer://
3. ftp://
4. gopher://
5. http://
6. https://
7. ldap://
8. mailto:
9. news:
10. nntp://

Mod_Rewrite Variables, from Source

• TIME - %04d%02d%02d%02d%02d%02d
• HTTPS - flag ? "on" : "off"
• TIME_DAY
• TIME_SEC
• TIME_MIN
• TIME_HOUR
• TIME_MON
• TIME_WDAY
• TIME_YEAR
• IS_SUBREQ - (main ? "true" : "false");
• PATH_INFO - path_info;
• AUTH_TYPE - ap_auth_type;
• HTTP_HOST - lookup_header("Host", ctx);
• SERVER_NAME - ap_get_server_name(r);
• REMOTE_ADDR - connection->remote_ip;
• SERVER_ADDR - connection->local_ip;
• HTTP_ACCEPT - lookup_header("Accept", ctx);
• THE_REQUEST - the_request;
• API_VERSION - "%d:%d",MODULE_MAGIC_NUMBER_MAJOR,MODULE_MAGIC_NUMBER_MINOR);
• HTTP_COOKIE - lookup_header("Cookie", ctx);
• SERVER_PORT - ap_get_server_port(r);
• REMOTE_HOST
• REMOTE_NAME, NULL);
• REMOTE_PORT - r->connection->remote_addr->port
• REMOTE_USER - user;
• SCRIPT_USER - "<unknown>";
• APR_FINFO_USER
• REQUEST_URI - uri;
• SCRIPT_GROUP - "<unknown>";
• REMOTE_IDENT - ap_get_remote_logname(r);
• HTTP_REFERER - lookup_header("Referer", ctx);
• QUERY_STRING - args;
• SERVER_ADMIN - server->server_admin;
• DOCUMENT_ROOT - ap_document_root(r);
• HTTP_FORWARDED - lookup_header("Forwarded", ctx);
• REQUEST_METHOD - method;
• HTTP_USER_AGENT - lookup_header("User-Agent", ctx);
• SCRIPT_FILENAME - same as request_filename
• REQUEST_FILENAME - same as script_filename
• SERVER_PROTOCOL - protocol
• SERVER_SOFTWARE - ap_get_server_banner();
• HTTP_PROXY_CONNECTION - lookup_header("Proxy-Connection", ctx);

REGEX Rewrite Guides

• Mod_Rewrite: Flags, Rules, Conditions, Tutorial and Variables
• Using Regular Expressions
• RewriteRule (mod_rewrite) guide

Mod_Rewrite Terms and Definitions

pattern

the RegExp pattern string

regexp

the RegExp pattern compilation

flags

Flags which control the substitution

forced_mimetype

forced MIME type of substitution

forced_handler

forced content handler of subst.

forced_responsecode

forced HTTP response status

env

added environment variables

cookie

added cookies

skip

number of next rules to skip

state

the RewriteEngine state

options

the RewriteOption state

rewritelogfile

the RewriteLog filename

rewritelogfp

the RewriteLog open filepointer

rewritelog: level

the RewriteLog level of verbosity

rewritemaps

the RewriteMap entries

rewriteconds

the RewriteCond entries (temp.)

rewriterules

the RewriteRule entries

directory

the directory where it applies

baseurl

the base-URL where it applies

Mod_Rewrite Errors

• Options FollowSymLinks or SymLinksIfOwnerMatch is off which implies that RewriteRule directive is forbidden: %s
• RewriteCond: bad argument line
• RewriteCond: NoCase option for non-regex pattern %s is not supported and will be ignored.
• RewriteCond: cannot compile regular expression
• RewriteRule: invalid HTTP response code %s for flag R
• RewriteRule: unknown flag
• RewriteRule: cannot compile regular expression
• RewriteOptions: MaxRedirects option has been removed in favor of the global LimitInternalRecursion directive and will be ignored.
• RewriteOptions: unknown option
• RewriteMap: bad path to txt map:
• RewriteMap: bad path to rnd map:
• RewriteMap: bad map:
• RewriteMap: bad path to dbm map:
• RewriteMap: dbm type
• RewriteMap: bad path to prg map:
• RewriteMap: internal map not found:
• RewriteMap: bad path to txt map:
• RewriteMap: file for map not found:
• Invalid RewriteLock path
• RewriteBase: only valid in per-directory config files
• RewriteBase: empty URL not allowed
• RewriteBase: argument is not a valid URL
• RewriteCond: bad flag delimiters
• RewriteCond: unknown flag
• RewriteLog and RewriteLogLevel are not supported by this build of mod_rewrite because it was compiled using the -DREWRITELOG_DISABLED compiler option. You have to recompile mod_rewrite WITHOUT this option in order to use the rewrite log.
• mod_rewrite: Invalid RewriteLog path %s
• mod_rewrite: could not open reliable pipe to RewriteLog filter %s
• mod_rewrite: Invalid RewriteLog path %s
• mod_rewrite: could not open RewriteLog file %s
• mod_rewrite: Running external rewrite maps without defining a RewriteLock is DANGEROUS!
• mod_rewrite: could not start RewriteMap program %s
• mod_rewrite: cant access text RewriteMap file %s
• mod_rewrite: cant access DBM RewriteMap file %s
• mod_rewrite: Parent could not create RewriteLock file %s
• mod_rewrite: Parent could not set permissions on RewriteLock check User and Group directives
• mod_rewrite: could not create rewrite_log_lock
• mod_rewrite: Could not set permissions on rewrite_log_lock check User and Group directives
• mod_rewrite: could not init rewrite_mapr_lock_acquire in child
• mod_rewrite: could not init rewrite log lock in child
• mod_rewrite: could not init map cache in child
• split uri=%s -> uri=%s, args=%s
• reduce %s -> %s
• strip matching prefix: %s -> %s
• add subst prefix: %s -> %s
• cant open RewriteMap file, see error log
• cache lookup FAILED, forcing new map lookup
• map lookup FAILED: map=%s[txt] key=%s
• map lookup OK: map=%s[txt] key=%s -> val=%s
• cache lookup OK: map=%s[txt] key=%s -> val=%s
• randomly chosen the subvalue `%s
• cant open DBM RewriteMap file, see error log
• cache lookup FAILED, forcing new map lookup
• map lookup FAILED: map=%s[dbm] key=%s
• map lookup OK: map=%s[dbm] key=%s -> val=%s
• cache lookup OK: map=%s[dbm] key=%s -> val=%s
• map lookup FAILED: map=%s key=%s
• map lookup OK: map=%s key=%s -> val=%s
• map lookup FAILED: map=%s key=%s
• map lookup OK: map=%s key=%s -> val=%s
• lookahead: path=%s var=%s -> val=%s
• lookahead: path=%s var=%s -> val=%s
• RESULT=%s
• escaping backreference %s to %s
• setting env variable %s to %s
• setting cookie %s, cookie
• skipping already set cookie %s
• RewriteCond URI (-U) check: path=%s -> status=%d
• RewriteCond file (-F) check: path=%s -> file=%s status=%d
• RewriteCond: input=%s pattern=%s%s%s%s => %s
• remember %s to have MIME-type %s
• remember %s to have Content-handler %s
• add path info postfix: %s -> %s%s
• strip per-dir prefix: %s -> %s
• applying pattern %s to uri %s
• rewrite %s -> %s, ctx->uri
• forcing responsecode %d for %s
• add per-dir prefix: %s -> %s%s
• forcing proxy-throughput with %s
• explicitly forcing redirect with %s
• implicitly forcing redirect (rc=%d) with %s
• forcing %s to get passed through to next API URI-to-filename handler
• init rewrite engine with requested uri %s
• go-ahead with proxy request %s [OK]
• dconf->directory,trying to replace prefix %s with %s
• escaping %s for redirect
• redirect to %s [REDIRECT/%d]
• initial URL equal rewritten URL: %s [IGNORING REWRITE]
• dconf->directory, trying to replace prefix %s with %s
• strip document_root prefix: %s -> %s
• internal redirect with %s [INTERNAL REDIRECT]
• pass through %s
• force filename %s to have MIME-type %s
• force filename %s to have the Content-handler %s,
• init rewrite engine with requested uri %s
• init rewrite engine with passed filename %s. Original uri = %s
• uri already rewritten. Status %s, Uri %s, %s
• attempt to make remote request from mod_rewrite without proxy enabled: %s
• go-ahead with proxy request %s [OK]
• escaping %s for redirect
• redirect to %s [REDIRECT/%d]
• local path result: %s
• prefixing with document_root of %s FAILED
• prefixed with document_root to %s
• go-ahead with %s [OK]
• pass through %s

URL Rewriting Module

This module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly.

It supports an unlimited number of additional rule conditions (which can operate on a lot of variables, even on HTTP headers) for granular matching and even external database lookups (either via plain text tables, DBM hash files or even external processes) for advanced URL substitution.

It operates on the full URLs (including the PATH_INFO part) both in per-server context (httpd.conf) and per-dir context (.htaccess) and even can generate QUERY_STRING parts on result. The rewriting result finally can lead to internal subprocessing, external request redirection or even to internal proxy throughput.

This module was originally written in April 1996 and gifted exclusively to the The Apache Software Foundation in July 1997 by

Ralf S. Engelschall rse [at] engenschall.com

Mod_Rewrite Variables Cheatsheet originally appeared on AskApache.com

I haven't had time to post much the past year, so I wanted to make up for that by publishing an article on a topic that would blow your mind and be something that you could actually start using and really get some benefit out of it. This is one of those articles that the majority of web hosting companies would love to see in paperback, so they could burn it. Now ask yourself, if a webhost makes money based on how much memory, bandwidth, and data used by a customer, what would they not want their customers to do? That's right, they do not want their customers to learn how to minimize and drastically reduce these moneymakers. They get giddy when you complain about slow-site-speed, or that it takes a long time for your site to load, because they have exactly the right answer- upgrade your memory, bandwidth, and data by purchasing a more expensive plan.

WARNING!! This article has some seriously advanced stuff in it, pretty far beyond my skill level as well (getting there). I personally shutdown some of my own servers with various webhosts because of this.. Note I said personally, not intentionally. Even after spending almost a year (this has been in my drafts folder a long time) using TMPFS on as many machines as I can, I still make mistakes (gotta pay attention!) and lose a tmpfs folder.. Oh and if you go experimenting with this stuff on your web host, you will almost definately, most certainly be on the road to getting your account terminated if you are with one of the cheap hosts. They hate this stuff because it cuts right into the heart of their profit curves and can seriously disrupt a poorly configured machine. DO NOT TRY THIS!! (except and of course on your own development machines). Of course the whole point of this article is how you can take advantage of this incredible filesystem to get crazy speed improvements.. Those are the follow up articles ;)

For those of you who thought modifying your server httpd.conf and htaccess files is very dangerous, you are right. But this is not like that, this is dangerous in the sense that if you try to rush through with your super amazing "copy and paste skills" (script kids) you will easily lose entire folders. That's because TMPFS is stored in RAM/Memory, and upon reboot RAM is cleared. I personally loathe disclaimers, and if you look around you will see there aren't many even with all my sloppy poorly documented articles... So be careful if you feel up to going further.

Introducing tmpfs

If I had to explain tmpfs in one breath, I'd say that tmpfs is like a ramdisk, but different. Like a ramdisk, tmpfs can use your RAM, but it can also use your swap devices for storage. And while a traditional ramdisk is a block device and requires a mkfs command of some kind before you can actually use it, tmpfs is a filesystem, not a block device; you just mount it, and it's there. All in all, this makes tmpfs the niftiest RAM-based filesystem I've had the opportunity to meet.

Beware of WebHosts

What is a modern day web hosting company? What costs do they actually have? A webhost's only unique ability is their connection to the Internet. That is why you can see such tremendous link speed. Other than that they consist of servers that are getting smaller and cheaper for them every month. The servers they use are generally just like any computer, except much larger and built specifically for multi-tasking.

Virtualization allows you to run multiple applications and operating systems independently on a single server. Additionally, administrators can quickly move workloads from one virtual workspace to another — easily prioritizing business needs while maximizing server resources....

Virtualization removes the limitations of the traditional IT approach, enabling a single PowerEdge server to operate multiple applications simultaneously in "virtual machines"

Hosting Company Tricks

Web hosts like to vaguely describe their products as if you are buying your own powerful machine, but in reality you get placed on the same machine as hundreds or thousands of other customers, and the server basically creates an operating system for each customer using virtualization technology. Everyone on the machine literally is sharing the same RAM and resources, many times even sharing IP address's, and the virtualization software lets them limit the amount of memory / cpu / disk / and bandwidth for each of these virtual machines. That is why so often when a web host has an outage they make big public announcements and it appears that hundreds or thousands of their customers have been affected.. One of their server farm machines goes offline and it literally takes down all the customers virtualized machines with it.

Why it gets Evil

Don't get me wrong, I absolutely love this technology, both the hardware virtualization and the software side, but what I truly do not appreciate is how these companies take advantage of their customers every day and know it. Here's what they do, they make justifications about why one plan costs more than another, and these justifications are always about the same thing: CPU's, how fast the data can crunch.. RAM/Memory: How fast and how much your server can handle in terms of traffic... Disk Usage: How much storage you have... And finally bandwidth: How fast can people get data off your sites, and how many people can connect.

Now lets think for a second. The webhost has a BIG computer/server/machine that has MASSIVE amounts of RAM, DISK, PROCESSING power, and NETWORK bandwidth.. but just like anything they all have limits. So if this machine has 10GB of RAM, and the webhost offered plans that have 1GB of RAM, then on that machine they can only have 10 customers right? WRONG. If each customer pays $100/month, then of course they would love to have as many customers on that machine as possible. This builtin incentive is just the reality and isn't anyone fault.

Where it gets Evil

Here's what goes on.. all the host advertises is the 1GB of guaranteed RAM with your machine, but for even if the web server was fairly busy it would never use all of that ram because all the software is careful not to use too much, or has no need for any RAM. Runtime libraries and internal caches use ram, but it's not directly accessed by the customer, only the software. What happens is when those 10 customers aren't using 100% of their ram, which never happens, then the virtualization technology can use that RAM elsewhere. So technically you do have 1GB of RAM available, but if you aren't using it then it is essentially FREE RAM that they can sell to another customer. The only way this wouldn't work of course is if all 11 customers somehow used 100% of RAM simultaneously, at that point the 11th customer would be ramless. But that is impossible because the system is a load-balancing system that provides both an upper and a lower limit to how much RAM is allotted to each virtual machine.

It sounds unrealistic but I see server farms all the time that are stuffed full of virtual machines, like situations where there are 100 1GB customers all sharing 10GB of RAM.. no-one uses the whole 1GB allotted to them as the maximum amount they can use, and they don't know because it appears they have a lot of free RAM, but really that is virtual RAM and could be used by anyone else on the machine.

Where it gets Fun (for me)

This is actually even worse for anyone who is using what they call "shared-hosting" which is the budget hosting that is the most common. With shared-hosting there is actually some skill involved on the hosting companies part, like real linux skills. In this setup they may or more often may not use any virtualization software. It's just a vanilla multi-user server machine where each customer gets a restricted unix account that powers their website using the same system as thousands of others on the box. This is usually dirt cheap because it costs so little to do, but alot of companies charge outrageous amounts for shared-hosting because they make it look really full-featured, which it can be, they just don't mention 1000 other people use the same machine, hard-drive, /tmp directory, network device, IP address, etc.. Alot of the times the cheaper end of the spectrum is where the most gifted system administrators are located, they are so good with linux administration that they could fit 10 customers and 100 websites on an XBOX converted to run linux, and you'd think you got a great deal until you found out! lol. Anyone alive is able to buy more hardware to expand their capacity to take on more customers, but it takes a lot of knowhow and real skill to have that many users on 1 machine. I've seen pretty extreme cases that are analogous to the XBOX example (which is possible by the way).

I personally love shared-hosting environments, because for those of us who know almost as much or more than the system administrators running the machine we are able to use a disproportionate (legally) amount of the CPU and RAM available on the system. So for example my sites would all show up fast and be able to handle more traffic than several other customers combined. Not because anything has been circumvented, but because I am able to access and utilize as much of the guaranteed 1GB of RAM that I am paying for every month, which is usually just a few bucks. The downside is that when you have corporate sites or really high-traffic sites then you are forced to move to a more powerful machine..

This leads to a familiar situation for some of you.. When your site starts becoming popular and you are getting a lot of traffic, this means that your site could be using 10x the amount of RAM and Bandwidth of any other customer in that server farm. And what that really means to the webhost is that you are costing them 10x what anyone else is.. And if they removed you, they would have the space for 10 new customers to take your place, and they would make 10x more money. DreamHost is notorious for terminating accounts because of that.. It happened to me except I was given the option to pay 5x more a month for their "upgrade" to a VPS. Giant shared-hosts advertise like crazy how they offer unlimited bandwidth, but when you start using 100x more bandwidth than anyone on your server you are costing them 100x what you are paying them, every month. That's why you will never see a webhost offering this kind of unlimited bandwidth that doesn't require you to sign a contract giving them permission to terminate your account for any reason. Seriously read the fine print at DreamHost or anywhere else, it's included because that is a core part of their business to terminate anyone using too much bandwidth since that is bandwidth they can't sell to dozens of other customers. That's why I eventually closed my account with them and moved to a legitimate company, it's a great host for spammers though.

Back in the mid-90's I was doing a lot of war-dialing with my modem and discovering all sorts of networks and machines, many of them were Unix and Solaris based public systems, and when I managed to gain access to the system and found myself staring at a unix shell I was very excited but also a total idiot. In those days of using the phone networks to research unknown systems it was very difficult for anyone to actually get the phone company to trace a call, so instead of what happens today where it is child's play to trace an IP address, back then it was a very real back-and-forth battle between the system admin and whoever was gaining access to their system. Essentially, I would gain a shell or some kind of terminal, and just go at it trying to figure out what it could do, trying all kinds of commands. Inevitably this would eventually alert even the laziest admin and they would proceed to attempt to lock me out. It was great sport and extremely addictive. When my favorite system (a massive sun machine in the basement of a big library) finally locked me out and I couldn't get back in I went to my local library and got some reading material -- one of my favorites was the red hat bible. I was able to acquire my own computer and the first thing I did was install red hat linux onto it from the discs included with the book. For the next several years I was essentially offline, all we had at home was a modem and it was becoming difficult to locate any more systems in my area code.. I was into phreaking of course as well, but I never was able to make free long-distance war-dialing a reality. So I just read the books and learned what I could. I would also goto the library when I could in order to use their machines which were connected to the internet (before aol it was much different than today's internet) and since my time was short I would download as many documents as I could so that I could read them offline. The TLDP documentation that we know today was around back then in various forms, and I read every HOWTO in the index, though not understanding half. The other big resource I found for really intense reading was the kernel documentation, which admitedly I still don't comprehend 1/4th of.. I try and peruse all the new documents when a new kernel is released, since the kernel is where all the real action is, hence the military authoritative name, and that is how I discovered one of the coolest features of Linux that I have found. TMPFS!

TMPFS kills the RAMDISK

Ok so we all know what RAM is, it's the memory cards that most people never see that is used by the computer to store and access data that all programs need. RAM is very expensive compared to most PC components, because it's what makes a computer blazing fast or slow. So real quick lets look at a few (there are not many) ways that various linux hackers use RAM in non-conventional ways in the past.

Tmpfs is a file system which keeps all files in virtual memory. Everything is temporary in the sense that no files will be created on your hard drive. If you reboot, everything in tmpfs will be lost.

In contrast to RAM disks, which get allocated a fixed amount of physical RAM, tmpfs grows and shrinks to accommodate the files it contains and is able to swap unneeded pages out to swap space.

Like a ramdisk, tmpfs can use your RAM, but it can also use your swap devices for storage. And while a traditional ramdisk is a block device and requires a mkfs command of some kind before you can actually use it, tmpfs is a filesystem, not a block device; you just mount it, and it's there. All in all, this makes tmpfs the niftiest RAM-based filesystem I've had the opportunity to meet.

If I had to explain tmpfs in one breath, I'd say that tmpfs is like a ramdisk, but different. Like a ramdisk, tmpfs can use your RAM, but it can also use your swap devices for storage. And while a traditional ramdisk is a block device and requires a mkfs command of some kind before you can actually use it, tmpfs is a filesystem, not a block device; you just mount it, and it's there. All in all, this makes tmpfs the niftiest RAM-based filesystem I've had the opportunity to meet.

What kind of filesystem is used on your server to store all your site files? EXT4, REISERFS, EXT3, NFS, etc.. are the usual filesystems, Windows users are limited to the NTFS filesystem. A filesystem is different than a device, a device is a hard-drive disk. A filesystem is how the device is formatted to allow for file and folder structures. A hard drive is slow compared to RAM, no question about that. So what if instead of your server serving files off a hard-drive it served files stored in RAM? 30x faster thats what happens!

I just figured out how to store my cached static files created by WP-Super Cache in my server's RAM, and the difference is unbelievable. My "AskApache Crazy Cache" plugin basically forces WP-Super Cache, Hyper Cache, etc.. to recreate a static cached file for every page on a blog. For the AskApache.com site this takes around 3 minutes to complete. Once I switched to using this new method of storing the files on RAM I am able to re-cache the entire site in about 15 seconds!!!!

tmpfs is a dynamically expandable/shrinkable ramdisk, and will # use almost no memory if not populated with files

Tmpfs is a file system which keeps all files in virtual memory.

Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost.

tmpfs puts everything into the kernel internal caches and grows and shrinks to accommodate the files it contains and is able to swap unneeded pages out to swap space. It has maximum size limits which can be adjusted on the fly via 'mount -o remount ...'

If you compare it to ramfs (which was the template to create tmpfs) you gain swapping and limit checking. Another similar thing is the RAM disk (/dev/ram*), which simulates a fixed size hard disk in physical RAM, where you have to create an ordinary filesystem on top. Ramdisks cannot swap and you do not have the possibility to resize them.

Since tmpfs lives completely in the page cache and on swap, all tmpfs pages currently in memory will show up as cached. It will not show up as shared or something like that. Further on you can check the actual RAM+swap use of a tmpfs instance with df(1) and du(1).

Both tmpfs and ramfs mount will give you the power of fast reading and writing files from and to the primary memory. When you test this on a small file, you may not see a huge difference. You’ll notice the difference only when you write large amount of data to a file with some other processing overhead such as network.

TMPFS uses RAM+SWAP

TMPFS is another filesystem with uniquely cool capabilities. It stores any files contained within it on RAM and in SWAP which means your server can access any files stored on TMPFS without even having to access the disk, which according to technical stats is around 30 times faster than accessing a file off disk.

Some other cool aspects of TMPFS are that it intelligently and automatically sizes itself to be just alittle bigger then it needs to be. So when you remove files to a folder stored on a TMPFS filesystem, the TMPFS filesystem shrinks by allocating less RAM and/or SWAP. Conversely when adding files to TMPFS it grows larger. You can set the max-size and max-number-of-files as a mount option to make sure your TMPFS never uses all of the available RAM and SWAP, which would halt your server.

Swap

Find the swap size.

# free -m -t
             total       used       free     shared    buffers     cached
Mem:           458         93        364          0          0          0
-/+ buffers/cache:         93        364
Swap:          900          0        900
Total:        1358         93       1264

Adding 3004144k swap on /dev/sdb2.  Priority:-1 extents:1 across:3004144k
Adding 2096472k swap on /dev/sda3.  Priority:-2 extents:1 across:2096472k

Using TMPFS for Cache

The method here will show how to create and use a TMPFS filesystem to hold all the static files created by WP-Super Cache. These static files are served to visitors instead of loading php for every request, so by moving those static files to TMPFS your server will be able to access and start sending your site to the browser 30x faster!

The WP-Super Cache plugin stores all the static files in the wp-content/cache folder of your WordPress installation, so to enable TMPFS we simply will create a new TMPFS filesystem and mount it to the wp-content/cache folder. That makes anything in that folder (all the static files) be part of the TMPFS filesystem.

Boosting Cache with TMPFS

There are a lot of maybe new concepts surrounding TMPFS and it may seem too complicated, but the process of actually setting up a robust tmpfs to use for wp-super-cache's cache folder is actually very simple. As long as you have shell access to your server and the permissions required (any sudo or private server should be good to go) you can set this up in a couple minutes and not really have to give it a second thought or debug anything. Here's the process I've used on several client sites.

1. Create a TMPFS Filesystem and Mount at /wp-content/cache/
2. Restore TMPFS Cached Files across Reboots
3. Keep a semi-current mirror of the TMPFS files on Disk

Create TMPFS at wp-content/cache

/etc/fstab

tmpfs /home/askapache/wp-content/cache tmpfs defaults,size=2g,noexec,nosuid,uid=648,gid=648,mode=1755 0 0

Restoring TMPFS across Reboots

In /etc/rc.local

ionice -c3 -n7 nice -n 19 rsync -ahv --stats --delete /_b/tmpfs/cache/ /home/askapache/wp-content/cache/ 1>/dev/null

Mirroring TMPFS to Disk

Cronjob entry

*/5 * * * * /usr/bin/ionice -c3 -n7 /bin/nice -n 19 /usr/bin/rsync -ah --stats --delete /home/askapache/wp-content/cache/ /_b/tmpfs/cache/ 1>/dev/null

/tmp, /var/run, and /var/lock

The directories /tmp, /var/run, and /var/lock contain files that are not needed across reboots. This means they are ideal candidates for tmpfs. HEre's how to do it.

tmpfs /var/run tmpfs defaults,rw,nosuid,mode=0755 0 0

tmpfs /var/lock tmpfs defaults,rw,noexec,nosuid,nodev,mode=1777 0 0

Resize /dev/shm

You can view your current /dev/shm size with the command df -ha|grep /dev/shm then if you want to resize that use the command:

mount -t tmpfs -o remount,size-2G,rw,nosuid,nodev tmpfs /dev/shm

Secure /dev/shm:
 
Step 1: Edit your /etc/fstab:
 
nano -w /etc/fstab
 
Locate:
 
none /dev/shm tmpfs defaults,rw 0 0
 
Change it to:
 
none /dev/shm tmpfs defaults,nosuid,noexec,rw 0 0
 
Step 2: Remount /dev/shm:
 
mount -o remount /dev/shm
 
guilt makes extensive use of the '$$' shell variable for temporary
files in /tmp. This is a serious security vulnerability; on multi-user
systems it allows an attacker to clobber files with something like the
following:
 
for i in `seq 1 32768`; do
ln -sf /etc/passwd /tmp/guilt.log.$i;
done
 
(In this example, if root does e.g. 'guilt push', /etc/passwd will get
clobbered.)

Securing and Using /tmp

• Secure temporary folders on existing Unix or Linux systems
• Encrypt Storage and Swap Space

tmpfs mount parameters

A good way to find a good tmpfs upper-bound is to use top to monitor your system's swap usage during peak usage periods. Then, make sure that you specify a tmpfs upper-bound that's slightly less than the sum of all free swap and free RAM during these peak usage times.

mode=1777 sets sticky bit on directory. Only file owners can delete files in this directory.

The following parameters accept a suffix k, m or g for Ki, Mi, Gi (binary kilo, mega and giga) and can be changed on remount.

• size: Override default maximum size of the filesystem. The size is given in bytes, and rounded down to entire pages. The default is half of the memory.The limit of allocated bytes for this tmpfs instance. The default is half of your physical RAM without swap. If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory.
• nr_inodes: Set number of inodes.
• nr_blocks: Set number of blocks.
• mode: The permissions as an octal number
• uid: The user id
• gid: The group id

mount -t tmpfs -o size=10G,nr_inodes=10k,mode=700 tmpfs /mytmpfs

Will give you tmpfs instance on /mytmpfs which can allocate 10GB RAM/SWAP in 10240 inodes and it is only accessible by root.

Using tmpfs for /tmp storage

Many users find it very convenient to use tmpfs for /tmp and /var/tmp which does a number of positive things. Any temporary files are instead created in RAM not your hard-drive, which means that reading/writing/accessing those temporary files by various processes doesn't slow down your hard-drive read/writes/accesses for your other processes. This also has a side-effect of making your hard-drive have a longer life as it reduces activity by a huge amount.

Remember that tmpfs uses both RAM and swap, so make sure your machine has a large swapfile, like gigabytes. If your tmpfs consumes all the swap and RAM then you are screwed, so make sure that you correctly set the mount options for the tmpfs so that it doesn't do that. If your /tmp or /var/tmp gets filled with tmp files that for some reason don't get deleted except at reboot, and your machine has a very high uptime, then you will want to run some cron jobs to periodically clean the /tmp and /var/tmp directories of older files...

Here's an example scenario: let's say that we have an existing filesystem mounted at /tmp. However, we decide that we'd like to start using tmpfs for /tmp storage.

with recent 2.4 kernels, you can mount your new /tmp filesystem without getting the "device is busy" error:

mount tmpfs /tmp -t tmpfs -o size=64m

With a single command, your new tmpfs /tmp filesystem is mounted at /tmp, on top of the already-mounted partition, which can no longer be directly accessed. However, while you can't get to the original /tmp, any processes that still have open files on this original filesystem can continue to access them. And, if you umount your tmpfs-based /tmp, your original mounted /tmp filesystem will reappear. In fact, you can mount any number of filesystems to the same mountpoint, and the mountpoint will act like a stack; unmount the current filesystem, and the last-most-recently mounted filesystem will reappear from underneath.

Bind Mounts

Using bind mounts, we can mount all, or even part of an already-mounted filesystem to another location, and have the filesystem accessible from both mountpoints at the same time!

For example, you can use bind mounts to mount your existing /tmp filesystem to /sites/askapache.com/tmp, as follows:

mount --bind /tmp /sites/askapache.com/tmp

Now, if you look inside /sites/askapache.com/tmp, you'll see your /tmp filesystem and all its files. And if you modify a file on your /tmp filesystem, you'll see the modifications in /sites/askapache.com/tmp as well. This is because they are one and the same filesystem; the kernel is simply mapping the filesystem to two different mountpoints for us.

Note that when you mount a filesystem somewhere else, any filesystems that were mounted to mountpoints inside the bind-mounted filesystem will not be moved along. In other words, if you have /tmp/cache on a separate filesystem, the bind mount we performed above will leave /sites/askapache.com/tmp/cache empty. You'll need an additional bind mount command to allow you to browse the contents of /tmp/cache at /sites/askapache.com/tmp/cache:

mount --bind /tmp/cache /sites/askapache.com/tmp/cache

Bind mounting and /dev/shm

glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for POSIX shared memory (shm_open, shm_unlink). Adding the following line to /etc/fstab should take care of this:

tmpfs  /dev/shm  tmpfs  defaults  0 0

Many systems by default have a tmpfs filesystem mounted at /dev/shm that defaults to a size of half of your physical RAM without swap. Say you decide that you'd like to start using tmpfs for /tmp, which currently lives on your root filesystem. Rather than mounting a new tmpfs filesystem to /tmp (which is possible), you may decide that you'd like the new /tmp to share the currently mounted /dev/shm filesystem. However, while you could bind mount /dev/shm to /tmp and be done with it, your /dev/shm contains some directories that you don't want to appear in /tmp. So, what do you do? How about this:

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

In this example, we first create a /dev/shm/tmp directory and then give it 1777 perms, the proper permissions for /tmp. Now that our directory is ready, we can mount /dev/shm/tmp, and only /dev/shm/tmp to /tmp. So, while /tmp/foo would map to /dev/shm/tmp/foo, there's no way for you to access the /dev/shm/bar file from /tmp.

/etc/default/tmpfs WorkAround

$ cat /etc/default/tmpfs
# SHM_SIZE sets the maximum size (in bytes) that the /dev/shm tmpfs can use.
# If this is not set then the size defaults to the value of TMPFS_SIZE
# if that is set; otherwise to the kernel's default.
#
# The size will be rounded down to a multiple of the page size, 4096 bytes.
SHM_SIZE=524288000
# TMPFS_SIZE sets the max size that /dev/shm can use.  By default, the
# kernel sets this upper limit to half of available memory.
TMPFS_SIZE=524288000

RSYNC vs. CP

rsync [options]  SRC DEST
rsync -av --delete --stats /home/wincom/public_html/wp-content/cache/ /backups/tmp-mnt/cache/
-a, --archive               archive mode; same as -rlptgoD (no -H)
-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks
-p, --perms                 preserve permissions
-t, --times                 preserve times
-g, --group                 preserve group
-o, --owner                 preserve owner (super-user only)
-D                          same as --devices --specials
    --devices               preserve device files (super-user only)
    --specials              preserve special files
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer

Mount Options

The following options apply to any file system that is being mounted (but not every file system actually honors them)

• async All I/O to the file system should be done asynchronously.
• atime Update inode access time for each access. This is the default.
• auto Can be mounted with the -a option.
• defaults Use default options: rw, suid, dev, exec, auto, nouser, and async.
• dev Interpret character or block special devices on the file system.
• exec Permit execution of binaries.
• group Allow an ordinary (i.e., non-root) user to mount the file system if one of his groups matches the group of the device. This option implies the options nosuid and nodev (unless overridden by subsequent options, as in the option line group,dev,suid).
• mand Allow mandatory locks on this filesystem. See fcntl(2).
• _netdev The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).
• noatime Do not update inode access times on this file system (e.g, for faster access on the news spool to speed up news servers).
• nodiratime Do not update directory inode access times on this filesystem.
• noauto Can only be mounted explicitly (i.e., the -a option will not cause the file system to be mounted).
• nodev Do not interpret character or block special devices on the file system.
• noexec Do not allow direct execution of any binaries on the mounted file system. (Until recently it was possible to run binaries anyway using a command like /lib/ld*.so /mnt/binary. This trick fails since Linux 2.4.25 / 2.6.0.)
• nomand Do not allow mandatory locks on this filesystem.
• nosuid Do not allow set-user-identifier or set-group-identifier bits to take effect. (This seems safe, but is in fact rather unsafe if you have suidperl(1) installed.)
• nouser Forbid an ordinary (i.e., non-root) user to mount the file system. This is the default.
• owner Allow an ordinary (i.e., non-root) user to mount the file system if he is the owner of the device. This option implies the options nosuid and nodev (unless overridden by subsequent options, as in the option line owner,dev,suid).
• remount Attempt to remount an already-mounted file system. This is commonly used to change the mount flags for a file system, especially to make a readonly file system writeable. It does not change device or mount point.
• ro Mount the file system read-only.
• _rnetdev Like _netdev, except "fsck -a" checks this filesystem during rc.sysinit.
• rw Mount the file system read-write.
• suid Allow set-user-identifier or set-group-identifier bits to take effect.
• sync All I/O to the file system should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.
• dirsync All directory updates within the file system should be done synchronously. This affects the following system calls: creat, link, unlink, symlink, mkdir, rmdir, mknod and rename.
• user Allow an ordinary user to mount the file system. The name of the mounting user is written to mtab so that he can unmount the file system again. This option implies the options noexec, nosuid, and nodev (unless overridden by subsequent options, as in the option line user,exec,dev,suid).
• users Allow every user to mount and unmount the file system. This option implies the options noexec, nosuid, and nodev (unless overridden by subsequent options, as in the option line users,exec,dev,suid).

Filesystems

You can find out what is filesystems are in place by using one of the following linux commands:

cat /etc/fstab
cat /etc/mtab
cat /proc/mounts
df -a

/etc/fstab

       /etc/fstab        file system table
       /etc/mtab         table of mounted file systems
       /etc/mtab~        lock file
       /etc/mtab.tmp     temporary file
       /etc/filesystems  a list of filesystem types to try

From /etc/mtab

none /tmp tmpfs size=128m,mode=1777 0 0

From /proc/mounts

none /tmp tmpfs rw,nodev,relatime,size=131072k 0 0

/etc/fstab

It is possible that files /etc/mtab and /proc/mounts don’t match. The first file is based only on the mount command options, but the content of the second file also depends on the kernel and others settings (e.g. remote NFS server. In particular case the mount command may reports unreliable information about a NFS mount point and the /proc/mounts file usually contains more reliable information.)

This file is used in three ways:

1. The following command (usually given in a bootscript) causes all file systems mentioned in fstab (of the proper type and/or having or not having the proper options) to be mounted as indicated, except for those whose line contains the noauto keyword. Adding the -F option will make mount fork, so that the filesystems are mounted simultaneously.

   mount -a [-t type] [-O optlist]

2. When mounting a file system mentioned in fstab, it suffices to give only the device, or only the mount point.
3. Normally, only the superuser can mount file systems. However, when fstab contains the user option on a line, anybody can mount the corresponding system.

The programs mount and umount maintain a list of currently mounted file systems in the file /etc/mtab.

Only the user that mounted a filesystem can unmount it again. If any user should be able to unmount, then use users instead of user in the fstab line. The owner option is similar to the user option, with the restriction that the user must be the owner of the special file. The group option is similar, with the restriction that the user must be member of the group of the special file.

The order of records in fstab is important because fsck(8), mount(8), and umount(8) sequentially iterate through fstab doing their thing.

The first field, (fs_spec)

Describes the block special device or remote filesystem to be mounted. For ordinary mounts it will hold (a link to) a block special device node (as created by mknod(8)) for the device to be mounted, like ‘/dev/cdrom’ or ‘/dev/sdb7’. For NFS mounts one will have <host>:<dir>, e.g., ‘knuth.aeb.nl:/’. For procfs, use ‘proc’.

Instead of giving the device explicitly, one may indicate the (ext2 or xfs) filesystem that is to be mounted by its UUID or volume label (cf. e2label(8) or xfs_admin(8)), writing LABEL= or UUID=, e.g., ‘LABEL=Boot’ or ‘UUID=3e6be9de-8139-11d1-9106-a43f08d823a6’. This will make the system more robust: adding or removing a SCSI disk changes the disk device name but not the filesystem volume label.

The second field, (fs_file)

Describes the mount point for the filesystem. For swap partitions, this field should be specified as ‘none’. If the name of the mount point contains spaces these can be escaped as ‘\040’.

The third field, (fs_vfstype), describes the type of the filesystem. Linux supports lots of filesystem types, such as adfs, affs, autofs, coda, coherent, cramfs, devpts, efs, ext2, ext3, hfs, hpfs, iso9660, jfs, minix, msdos, ncpfs, nfs, ntfs, proc, qnx4, reiserfs, romfs, smbfs, sysv, tmpfs, udf, ufs, umsdos, vfat, xenix, xfs, and possibly others. For more details, see mount(8). For the filesystems currently supported by the running kernel, see /proc/filesystems. An entry swap denotes a file or partition to be used for swapping, cf. swapon(8). An entry ignore causes the line to be ignored. This is useful to show disk partitions which are currently unused.

The fourth field, (fs_mntops)

Describes the mount options associated with the filesystem. It is formatted as a comma separated list of options. It contains at least the type of mount plus any additional options appropriate to the filesystem type. For documentation on the available options for non-nfs file systems, see mount(8). For documentation on all nfs-specific options have a look at nfs(5).

Common for all types of file system are the options:

• noauto: (do not mount when "mount -a" is given, e.g., at boot time)
• user: (allow a user to mount)
• owner: (allow device owner to mount)
• pamconsole: (allow a user at the console to mount)
• comment: (e.g., for use by fstab-maintaining programs).

The fifth field, (fs_freq)

Used for these filesystems by the dump(8) command to determine which filesystems need to be dumped. If the fifth field is not present, a value of zero is returned and dump will assume that the filesystem does not need to be dumped.

The sixth field, (fs_passno)

Used by the fsck(8) program to determine the order in which filesystem checks are done at reboot time. The root filesystem should be specified with a fs_passno of 1, and other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. If the sixth field is not present or zero, a value of zero is returned and fsck will assume that the filesystem does not need to be checked.

More Reading

• Overview of RAMFS and TMPFS on Linux
• ramfs, rootfs and initramfs
• Tmpfs is a file system which keeps all files in virtual memory
• IBM: Advanced filesystem implementor's guide, Part 3
• TMPFS Wikipedia Entry
• Shared Memory
• Create turbocharged storage using tmpfs
• Where MySQL Stores Temporary Files
• speeding up firefox with tmpfs and automatic rsync (shell-script) Original
• kernel documentation for tmpfs
• initscripts: please don't mount /dev/shm noexec
• HOWTO: Using tmpfs for /tmp, /var/{log,run,lock...}
• Gentoo Forums: Using tmpfs for /var/{log,lock,...}
• [TIP] Firefox and tmpfs: a surprising improvement

Experiment: MySQL tmpdir on tmpfs

In MySQL, the tmpdir path is mainly used for disk-based sorts (if the sort_buffer_size is not enough) and disk-based temp tables. The latter cannot always be avoided even if you made tmp_table_size and max_heap_table_size quite large, since MEMORY tables don’t support TEXT/BLOB type columns, and also since you just really don’t want to run the risk of exceeding available memory by setting these things too large.

Use tmpfs for MySQL

--tmpdir=path, -t path

The path of the directory to use for creating temporary files. It might be useful if your default /tmp directory resides on a partition that is too small to hold temporary tables. Starting from MySQL 4.1.0, this option accepts several paths that are used in round-robin fashion. Paths should be separated by colon characters (“:”) on Unix and semicolon characters (“;”) on Windows, NetWare, and OS/2. If the MySQL server is acting as a replication slave, you should not set --tmpdir to point to a directory on a memory-based file system or to a directory that is cleared when the server host restarts. For more information about the storage location of temporary files, see Section A.1.4.4, “Where MySQL Stores Temporary Files”. A replication slave needs some of its temporary files to survive a machine restart so that it can replicate temporary tables or LOAD DATA INFILE operations. If files in the temporary file directory are lost when the server restarts, replication fails.

On Unix, MySQL uses the value of the TMPDIR environment variable as the path name of the directory in which to store temporary files. If TMPDIR is not set, MySQL uses the system default, which is usually /tmp, /var/tmp, or /usr/tmp. If the file system containing your temporary file directory is too small, you can use the --tmpdir option to mysqld to specify a directory in a file system where you have enough space. Starting from MySQL 4.1, the --tmpdir option can be set to a list of several paths that are used in round-robin fashion. Paths should be separated by colon characters (“:”) on Unix and semicolon characters (“;”) on Windows, NetWare, and OS/2. Note To spread the load effectively, these paths should be located on different physical disks, not different partitions of the same disk. If the MySQL server is acting as a replication slave, you should not set --tmpdir to point to a directory on a memory-based file system or to a directory that is cleared when the server host restarts. A replication slave needs some of its temporary files to survive a machine restart so that it can replicate temporary tables or LOAD DATA INFILE operations. If files in the temporary file directory are lost when the server restarts, replication fails. MySQL creates all temporary files as hidden files. This ensures that the temporary files are removed if mysqld is terminated. The disadvantage of using hidden files is that you do not see a big temporary file that fills up the file system in which the temporary file directory is located.

Shell Script for Firefox tmpfs

#!/bin/bash
### Bind temporary directories to /dev/shm ###
# I do this instead of mounting tmpfs on the #
# directories, so less memory gets wasted.   #
##############################################
mkdir /dev/shm/{tmp,lock}
mount --bind /dev/shm/tmp /tmp
mount --bind /dev/shm/tmp /var/tmp
mount --bind /dev/shm/lock /var/lock
chmod 1777 /dev/shm/{tmp,lock}

Hey! You made it!@ at least to the bottom of the page.. I still have to finish this article, so check back in a few months.

30x Faster Cache and Site Speed with TMPFS originally appeared on AskApache.com

My traffic is growing, alot, and I need to plan how I'm going to maintain scalability, high availability, and redundancy. Scalability is an application's ability to support a growing number of users. High availability can be defined as redundancy or speed. I decided to setup Round Robin DNS for static.askapache.com, which is the "static" subdomain of AskApache that serves all the static assets like images, javascript, css, etc.. (BTW, the z stands for ZAP).. All I needed to attempt setting this up was another hosting account on a separate server. I have hosting accounts with around 10 different companies from working with various clients over the years, like Powweb and I don't use them because they suck in terms of the unix environment. Many of these web hosts are actually very fast bandwidth-wise..

Round Robin Concept

A few months ago I was given a free hosting account on HostGator to evaluate and tempt me away from DreamHost to become a Gator. I get a lot of these types of offers from time to time from the smaller Web companies who read AskApache.. but when I noticed the SSH access was jailed and saw how restrictive the shell was I felt like I was on a windows box.. I want a shell, cpanel sucks. I compile and run everything from the shell so thats was a deal-breaker and I sorta forgot all about it.

The goal is to add the HostGator server to be an exact mirror of the static.askapache.com domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.

Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)

DNS - Domain Name System

The Domain Name System is often analogized as a "phone book" for the Internet because it translates human-friendly computer hostnames into IP addresses. For example, www.askapache.com translates to 208.113.134.190. Every request for a human-friendly hostname first needs to be translated to the IP before the server can be queried. Normally each hostname corresponds to exactly 1 IP address. But in a Round Robin setup, the idea is to have the hostname correspond to multiple IP addresses, which are different servers that house the exact same content, resulting in some hosts requesting files from one server, and another host requesting files from the other server, resulting in less CPU resources and bandwidth.

Here is an the zone for static.askapache.com Round Robin records:

Round Robin DNS

Round robin DNS is a technique of load distribution, load balancing, or fault-tolerance provisioning multiple, redundant Internet Protocol service hosts, e.g., Web servers, FTP servers, by managing the Domain Name System's (DNS) responses to address requests from client computers according to an appropriate statistical model.

In its simplest implementation Round-robin DNS works by responding to DNS requests not only with a single IP address, but a list of IP addresses of several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query so that on different connection attempts clients would receive service from different providers, thus distributing the overall load among servers.

Round robin DNS is often used for balancing the load of geographically-distributed Web servers. For example, a company has one domain name and three identical web sites residing on three servers with three different IP addresses. When one user accesses the home page it will be sent to the first IP address. The second user who accesses the home page will be sent to the next IP address, and the third user will be sent to the third IP address. In each case, once the IP address is given out, it goes to the end of the list. The fourth user, therefore, will be sent to the first IP address, and so forth.

Although easy to implement, round robin DNS has problematic drawbacks, such as those arising from record caching in the DNS hierarchy itself, as well as client-side address caching and reuse, the combination of which can be difficult to manage. Round robin DNS should not solely be relied upon for service availability. If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service.

Does Round Robin Work

Definately. I can look at the access logs for both servers and see that they are splitting the requests nicely. It is definately not an exact split however, look at these statistics and you'll see what I mean.

$ dig @ns1.dreamhost.com +authority +all static.askapache.com
 
;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       69.56.174.114
static.askapache.com.        14400   IN      A       64.111.114.111
 
$ dig @ns1.dreamhost.com +authority +all static.askapache.com
 
;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       64.111.114.111
static.askapache.com.        14400   IN      A       69.56.174.114

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used which has the effect of significantly increasing the load on the DNS (and is not always implemented consistently). In this case the cure may be worse than the disease Good news we have good load balancing on our web servers. Bad news we need 17 more DNS servers!. Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static list, loads should be reasonably balanced .

Full root server query

Tracing to static.askapache.com[a] via A.ROOT-SERVERS.NET, maximum of 3 retries
A.ROOT-SERVERS.NET [.] (198.41.0.4)
 |\___ L.GTLD-SERVERS.NET [com] (192.41.162.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ D.GTLD-SERVERS.NET [com] (192.31.80.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ J.GTLD-SERVERS.NET [com] (192.48.79.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ H.GTLD-SERVERS.NET [com] (192.54.112.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ E.GTLD-SERVERS.NET [com] (192.12.94.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ F.GTLD-SERVERS.NET [com] (192.35.51.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ I.GTLD-SERVERS.NET [com] (192.43.172.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ G.GTLD-SERVERS.NET [com] (192.42.93.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ B.GTLD-SERVERS.NET [com] (2001:0503:231d:0000:0000:0000:0002:0030) Not queried
 |\___ B.GTLD-SERVERS.NET [com] (192.33.14.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ A.GTLD-SERVERS.NET [com] (2001:0503:a83e:0000:0000:0000:0002:0030) Not queried
 |\___ A.GTLD-SERVERS.NET [com] (192.5.6.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ C.GTLD-SERVERS.NET [com] (192.26.92.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ M.GTLD-SERVERS.NET [com] (192.55.83.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
  \___ K.GTLD-SERVERS.NET [com] (192.52.178.30)
       |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
       |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
        \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114

Live Online DNS Tools

• DNSstuff.com - Your Destination for DNS and Networking Tools
• ZoneCheck
• Quick Check - Pingability.com
• Squishywishywoo: complete dns traversal checking
• ZoneCut DNS
• DNS Sleuth
• DNS test - Domain name DNS test - pweb.cz
• OpenDNS - Support - CacheCheck
• DNSReport
• DNSTrace
• intoDNS
• DNSthru.com
• DNS Doctor
• Power Check

More Reading

• HOWTO - Configure Load Balancing
• Load Sharing with DNS

RFC's related to DNS

• RFC 920: Specified original TLDs: .arpa, .com, .edu, .org, .gov, .mil and two-character country codes
• RFC 1032: Domain administrators guide
• RFC 1033: Domain administrators operations guide
• RFC 1034: Domain Names - Concepts and Facilities.
• RFC 1035: Domain Names - Implementation and Specification
• RFC 1101: DNS Encodings of Network Names and Other Types
• RFC 1123: Requirements for Internet Hosts -- Application and Support
• RFC 1912: Common DNS Operational and Configuration Errors
• RFC 1995: Incremental Zone Transfer in DNS
• RFC 1996: A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)
• RFC 2136: Dynamic Updates in the domain name system (DNS UPDATE)
• RFC 2181: Clarifications to the DNS Specification
• RFC 2182: Selection and Operation of Secondary DNS Servers
• RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
• RFC 2317: Classless IN-ADDR.ARPA delegation
• RFC 2671: Extension Mechanisms for DNS (EDNS0)
• RFC 3597: Handling of Unknown DNS Resource Record (RR) Types
• RFC 3696: Application Techniques for Checking and Transformation of Names
• RFC 4343: Domain Name System (DNS) Case Insensitivity Clarification
• RFC 4592: The Role of Wildcards in the Domain Name System
• RFC 4892: Requirements for a Mechanism Identifying a Name Server Instance
• RFC 5001: DNS Name Server Identifier Option (NSID)
• RFC 5395: Domain Name System (DNS) IANA Considerations

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Table of Contents

• Round Robin Concept
• DNS - Domain Name System
• Round Robin DNS
• Efficacy of DNS Load Balancing
• Live Online DNS Tools
• More Reading
• RFC's related to DNS

DNS Round Robin Configuration using Rsync over SSH originally appeared on AskApache.com