Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I’m not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn’t it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won’t be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU’s handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore “processes”) are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place… because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I’ve opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that’s completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 – tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching – CPU processing
2. DISK IO Bottleneck – Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 – Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it’s important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N – add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not “punchy” enough), the scheduler was decoupled from ‘time slice’ and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization “split” between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be “multiplicative” (or exponential) – that way it does not matter which nice level you start out from, the ‘relative result’ will always be the same.

The third complaint (of negative nice levels not being “punchy” enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID’s 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following “real-time” policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following “normal” (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload’s tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn’t set a specific io priority. This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it’s directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn’t really apply here.

ionice

ionice – get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c – The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n – The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p – Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs ‘bash’ as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]

-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities – schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces — thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I’ve found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn’t be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that “bonds” a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit – get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell’s resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft’ resource limit

-H

use the `hard’ resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process’s data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft’, `hard’, and `unlimited’ stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process’s virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process’s nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 – rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process’s resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler’s nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on AskApache.

The Worst Kind of People

Spammers, and Leechers. They operate like this: Let’s say you have some mp3 files on a server, and SOMEWHERE on the web there is a link to that mp3 file’s location. This includes in javascript files, css files, robots.txt files, the spammers and leechers robots check all those files looking for the type of link they are looking for. Then they try to request that file usually utilizing a number of different types of requests to get access to the file. Then they use it for personal gain, at your peril.

Some robots perform valuable services for the world wide web community, and other leeching programming is pretty cool, so not all these activities are perpetrated by nefarious spammers.

Blocking by IP COOKIE

Ok so if a link exists to your file, it is going to be requested by a robot eventually, so the way to defeat them is by doing something on your site that modifies the way a user would request it. Robots for the most part are not javascript-capable, so the most-common advanced method is to set a cookie using javascript, and then we can check for that cookie in the request for the file using mod_rewrite.

So if your site sets a cookie named fspammers, and furthermore gives that cookie a value of 445, then this is what the request sent by an HTTP Client like Firefox looks like.

GET /hotlink/lovefreedom.mp3 HTTP/1.1
Host: z.askapache.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.askapache.com/wordpress/seo-in-wordpress.html
Cookie: fspammers=455

Mod_Rewrite HTTP Headers

The mod_rewrite module has access to ALL the HTTP Headers sent in a request, so for each of the HTTP Headers in the request example above, we can use mod_rewrite to validate.

Mod_Rewrite .htaccess Example

Finally, now that everyone is on the same page about what is really going on, here is the .htaccess code that blocks any requests for anything in the /hotlink/ folder.

Here are the triggers this code blocks access based on.

1. Cookie: Checks if fspammers cookie is present, and that it has the value of 445.
2. HTTP Protocol: Checks if HTTP 1.1 is being used (many robots use 1.0)
3. Host: Checks that the HOST Requested was z.askapache.com
4. Referer: Checks for Referring site is z.askapache.com or www.askapache.com

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_COOKIE} !^.*fspammers=445.*$ [NC,OR]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /(.*)\ HTTP/1\.1 [NC,OR]
RewriteCond %{HTTP_HOST} !^z\.askapache\.com$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^http://(www|z)\.askapache\.com.*$ [NC]
RewriteRule ^hotlink/.*$ - [F]</p>

Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking originally appeared on AskApache.

The following undocumented techniques and methods will allow you to utilize mod_rewrite at an “expert level” by showing you how to unlock its secrets.

Most if not all web developers and server administrators struggle with Apache mod_rewrite. It’s very tough and only gets a little easier with practice. Until Now! Get ready to explode your learning curve, I figured something out.

Why mod_rewrite is so tough

I have come to the conclusion, after many hours of zenful thought, that the reason mod_rewrite is so tough is pretty obvious, people are trying to apply regular-expressions to URLs and Variables that they don’t really understand. They understand what they want, but they don’t understand what the URLS and Variables are that they are trying to rewrite.

Hit-Or-Miss with mod_rewrite

A lot of the mod_rewrite “experts” and “gurus” floating around the net absolutely know their mod_rewrite, but what separates them from a beginner or novice is for the most part an understanding of what the URLS and Variables look like that are targeted by the regular expressions. Take this simple rewriterule that rewrites requests made without the www to www.

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule .+ http://www.askapache.com%{REQUEST_URI}

Pretty simple right? WRONG. Most people could not figure that out..

Why?

The reason intelligent people can’t figure that out is because they have no idea what HTTP_HOST or REQUEST_URI actually looks like. How can you write a rule for something if you don’t know what it looks like? You can’t.

When Not To Use Mod_Rewrite

Ok so heres an important concept that alot of people haven’t heard. You should only use mod_rewrite’s rewriterule when you use a rewritecond or if you are rewriting internally like my feedcount hack.

If you are simply redirecting one url to another, you should definately be using the much easier mod_alias’s redirect and redirectmatch, which is enabled on most Apache servers.

When To Use Mod_Rewrite

So then, you should only use mod_rewrite’s rewriterule when you are checking against one of the Apache Environment Variables to determine whether to rewrite or not. This is where the Apache Documentation is grossly lacking. They don’t tell you what those variables look like, leaving us completely incapable of creating rewrites based on them. Not anymore.

Mod_Rewrite Environment Variables (The Secret)

Here’s the variables I have found accessible by mod_rewrite (both documented and undocumented). A thing to note is that you can set these variables early in an .htaccess file using SetEnv, RewriteRule, Header, etc.. and they will be accessible at the end of the .htaccess file.

• API_VERSION
• AUTH_TYPE
• CONTENT_LENGTH
• CONTENT_TYPE
• DOCUMENT_ROOT
• GATEWAY_INTERFACE
• HTTPS
• HTTP_ACCEPT
• HTTP_ACCEPT_CHARSET
• HTTP_ACCEPT_ENCODING
• HTTP_ACCEPT_LANGUAGE
• HTTP_CACHE_CONTROL

• HTTP_CONNECTION
• HTTP_COOKIE
• HTTP_FORWARDED
• HTTP_HOST
• HTTP_KEEP_ALIVE
• HTTP_PROXY_CONNECTION
• HTTP_REFERER
• HTTP_USER_AGENT
• IS_SUBREQ
• ORIG_PATH_INFO
• ORIG_PATH_TRANSLATED
• ORIG_SCRIPT_FILENAME

• ORIG_SCRIPT_NAME
• PATH
• PATH_INFO
• PHP_SELF
• QUERY_STRING
• REDIRECT_QUERY_STRING
• REDIRECT_REMOTE_USER
• REDIRECT_STATUS
• REDIRECT_URL
• REMOTE_ADDR
• REMOTE_HOST
• REMOTE_IDENT

• REMOTE_PORT
• REMOTE_USER
• REQUEST_FILENAME
• REQUEST_METHOD
• REQUEST_TIME
• REQUEST_URI
• SCRIPT_FILENAME
• SCRIPT_GROUP
• SCRIPT_NAME
• SCRIPT_URI
• SCRIPT_URL
• SCRIPT_USER

• SERVER_ADDR
• SERVER_ADMIN
• SERVER_NAME
• SERVER_PORT
• SERVER_PROTOCOL
• SERVER_SIGNATURE
• SERVER_SOFTWARE
• THE_REQUEST
• TIME
• TIME_DAY
• TIME_HOUR
• TIME_MIN

• TIME_MON
• TIME_SEC
• TIME_WDAY
• TIME_YEAR
• TZ
• UNIQUE_ID

Decoding Mod_Rewrite Variables

So when I realized my problem was that I didn’t know the value of the variable being tested by the RewriteCond, I set out to try and discover how to view those variables.. Keep in mind you can also use RewriteLogging, but its only allowed for root users who can edit the httpd.conf, this is .htaccess.

Setting Environment Variables with RewriteRule

I discovered a multitude of methods to set and view apache environment variables, using various modules and some core tricks, but the method that allows me to view the most environment variables is RewriteRule.. I wanted to use SetEnvIf more, but its just not as powerful as mod_rewrite, due to programming.

This code sets the variable INFO_REQUEST_URI to have the value of REQUEST_URI.

RewriteEngine On
RewriteBase /
RewriteRule .* - [E=INFO_REQUEST_URI:%{REQUEST_URI},NE]

Saving the Apache Variable Values

Now the trick is how to view that environment variable… The method I came up with is nice… We will send the environment variable value in an HTTP Header, as there isn’t much data manipulation/validation so you get an accurate look at the actual value.. At first I tried adding the variable value to a redirection using the query_string.. but a HTTP_USER_AGENT value doesn’t play well as a query_string.

Using RequestHeader in .htaccess

This code takes advantage of the incredible mod_headers apache module to actually ADD a whole new header to YOUR request. Seriously one of the coolest tricks I’ve found yet.. Its almost the same as being able to spoof POST requests! Since Headers can be protected data… especially the HTTP_COOKIE header..

RequestHeader set INFO_REQUEST_URI "%{INFO_REQUEST_URI}e"

Viewing the Variable Values

Now you can use any kind of server-run interpreter like perl, php, ruby, etc., to view all the variable values. All cgi-script handlers like those are able to view request headers..

PHP Code to access Apache Variables

Works even in safe-mode… any interpreter can view HTTP Headers! Note that each of these variables are added as HTTP headers to the request for the script.. kinda confusing.. So each variable sent as a header is prefixed with HTTP_ to denote it was a header.

<?php
header("Content-Type: text/plain");
$INFO=$MISS=array();
foreach($_SERVER as $v=>$r)
{
  if(substr($v,0,9)=='HTTP_INFO')
  {
    if(!empty($r))$INFO[substr($v,10)]=$r;
    else $MISS[substr($v,10)]=$r;
  }
}
 
/* thanks Mike! */
ksort($INFO);
ksort($MISS);
ksort($_SERVER);
 
echo "Received These Variables:\n";
print_r($INFO);
 
echo "Missed These Variables:\n";
print_r($MISS);
 
echo "ALL Variables:\n";
print_r($_SERVER);
?>

Time to Get Crazy

Just create the above php file on your site as /test/index.php or whatever, then create /test/.htaccess which should contain the below .htaccess file snippet. Now just request /test/index.php and be amazed! If you’re looking for more general help check out this excellent mod_rewrite cheat sheet.

Ok, so I’ve prepared the .htaccess code you can use to view the values of all these variables. Just add it to a .htaccess file and make a request. For this test I created an index.php file that printed out all the $_SERVER variables, and made requests to it.

RewriteEngine On
RewriteBase /
RewriteRule .* - [E=INFO_API_VERSION:%{API_VERSION},NE]
RewriteRule .* - [E=INFO_AUTH_TYPE:%{AUTH_TYPE},NE]
RewriteRule .* - [E=INFO_CONTENT_LENGTH:%{CONTENT_LENGTH},NE]
RewriteRule .* - [E=INFO_CONTENT_TYPE:%{CONTENT_TYPE},NE]
RewriteRule .* - [E=INFO_DOCUMENT_ROOT:%{DOCUMENT_ROOT},NE]
RewriteRule .* - [E=INFO_GATEWAY_INTERFACE:%{GATEWAY_INTERFACE},NE]
RewriteRule .* - [E=INFO_HTTPS:%{HTTPS},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT:%{HTTP_ACCEPT},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_CHARSET:%{HTTP_ACCEPT_CHARSET},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_ENCODING:%{HTTP_ACCEPT_ENCODING},NE]
RewriteRule .* - [E=INFO_HTTP_ACCEPT_LANGUAGE:%{HTTP_ACCEPT_LANGUAGE},NE]
RewriteRule .* - [E=INFO_HTTP_CACHE_CONTROL:%{HTTP_CACHE_CONTROL},NE]
RewriteRule .* - [E=INFO_HTTP_CONNECTION:%{HTTP_CONNECTION},NE]
RewriteRule .* - [E=INFO_HTTP_COOKIE:%{HTTP_COOKIE},NE]
RewriteRule .* - [E=INFO_HTTP_FORWARDED:%{HTTP_FORWARDED},NE]
RewriteRule .* - [E=INFO_HTTP_HOST:%{HTTP_HOST},NE]
RewriteRule .* - [E=INFO_HTTP_KEEP_ALIVE:%{HTTP_KEEP_ALIVE},NE]
RewriteRule .* - [E=INFO_HTTP_MOD_SECURITY_MESSAGE:%{HTTP_MOD_SECURITY_MESSAGE},NE]
RewriteRule .* - [E=INFO_HTTP_PROXY_CONNECTION:%{HTTP_PROXY_CONNECTION},NE]
RewriteRule .* - [E=INFO_HTTP_REFERER:%{HTTP_REFERER},NE]
RewriteRule .* - [E=INFO_HTTP_USER_AGENT:%{HTTP_USER_AGENT},NE]
RewriteRule .* - [E=INFO_IS_SUBREQ:%{IS_SUBREQ},NE]
RewriteRule .* - [E=INFO_ORIG_PATH_INFO:%{ORIG_PATH_INFO},NE]
RewriteRule .* - [E=INFO_ORIG_PATH_TRANSLATED:%{ORIG_PATH_TRANSLATED},NE]
RewriteRule .* - [E=INFO_ORIG_SCRIPT_FILENAME:%{ORIG_SCRIPT_FILENAME},NE]
RewriteRule .* - [E=INFO_ORIG_SCRIPT_NAME:%{ORIG_SCRIPT_NAME},NE]
RewriteRule .* - [E=INFO_PATH:%{PATH},NE]
RewriteRule .* - [E=INFO_PATH_INFO:%{PATH_INFO},NE]
RewriteRule .* - [E=INFO_PHP_SELF:%{PHP_SELF},NE]
RewriteRule .* - [E=INFO_QUERY_STRING:%{QUERY_STRING},NE]
RewriteRule .* - [E=INFO_REDIRECT_QUERY_STRING:%{REDIRECT_QUERY_STRING},NE]
RewriteRule .* - [E=INFO_REDIRECT_REMOTE_USER:%{REDIRECT_REMOTE_USER},NE]
RewriteRule .* - [E=INFO_REDIRECT_STATUS:%{REDIRECT_STATUS},NE]
RewriteRule .* - [E=INFO_REDIRECT_URL:%{REDIRECT_URL},NE]
RewriteRule .* - [E=INFO_REMOTE_ADDR:%{REMOTE_ADDR},NE]
RewriteRule .* - [E=INFO_REMOTE_HOST:%{REMOTE_HOST},NE]
RewriteRule .* - [E=INFO_REMOTE_IDENT:%{REMOTE_IDENT},NE]
RewriteRule .* - [E=INFO_REMOTE_PORT:%{REMOTE_PORT},NE]
RewriteRule .* - [E=INFO_REMOTE_USER:%{REMOTE_USER},NE]
RewriteRule .* - [E=INFO_REQUEST_FILENAME:%{REQUEST_FILENAME},NE]
RewriteRule .* - [E=INFO_REQUEST_METHOD:%{REQUEST_METHOD},NE]
RewriteRule .* - [E=INFO_REQUEST_TIME:%{REQUEST_TIME},NE]
RewriteRule .* - [E=INFO_REQUEST_URI:%{REQUEST_URI},NE]
RewriteRule .* - [E=INFO_SCRIPT_FILENAME:%{SCRIPT_FILENAME},NE]
RewriteRule .* - [E=INFO_SCRIPT_GROUP:%{SCRIPT_GROUP},NE]
RewriteRule .* - [E=INFO_SCRIPT_NAME:%{SCRIPT_NAME},NE]
RewriteRule .* - [E=INFO_SCRIPT_URI:%{SCRIPT_URI},NE]
RewriteRule .* - [E=INFO_SCRIPT_URL:%{SCRIPT_URL},NE]
RewriteRule .* - [E=INFO_SCRIPT_USER:%{SCRIPT_USER},NE]
RewriteRule .* - [E=INFO_SERVER_ADDR:%{SERVER_ADDR},NE]
RewriteRule .* - [E=INFO_SERVER_ADMIN:%{SERVER_ADMIN},NE]
RewriteRule .* - [E=INFO_SERVER_NAME:%{SERVER_NAME},NE]
RewriteRule .* - [E=INFO_SERVER_PORT:%{SERVER_PORT},NE]
RewriteRule .* - [E=INFO_SERVER_PROTOCOL:%{SERVER_PROTOCOL},NE]
RewriteRule .* - [E=INFO_SERVER_SIGNATURE:%{SERVER_SIGNATURE},NE]
RewriteRule .* - [E=INFO_SERVER_SOFTWARE:%{SERVER_SOFTWARE},NE]
RewriteRule .* - [E=INFO_THE_REQUEST:%{THE_REQUEST},NE]
RewriteRule .* - [E=INFO_TIME:%{TIME},NE]
RewriteRule .* - [E=INFO_TIME_DAY:%{TIME_DAY},NE]
RewriteRule .* - [E=INFO_TIME_HOUR:%{TIME_HOUR},NE]
RewriteRule .* - [E=INFO_TIME_MIN:%{TIME_MIN},NE]
RewriteRule .* - [E=INFO_TIME_MON:%{TIME_MON},NE]
RewriteRule .* - [E=INFO_TIME_SEC:%{TIME_SEC},NE]
RewriteRule .* - [E=INFO_TIME_WDAY:%{TIME_WDAY},NE]
RewriteRule .* - [E=INFO_TIME_YEAR:%{TIME_YEAR},NE]
RewriteRule .* - [E=INFO_TZ:%{TZ},NE]
RewriteRule .* - [E=INFO_UNIQUE_ID:%{UNIQUE_ID},NE]
 
RequestHeader set INFO_API_VERSION "%{INFO_API_VERSION}e"
RequestHeader set INFO_AUTH_TYPE "%{INFO_AUTH_TYPE}e"
RequestHeader set INFO_CONTENT_LENGTH "%{INFO_CONTENT_LENGTH}e"
RequestHeader set INFO_CONTENT_TYPE "%{INFO_CONTENT_TYPE}e"
RequestHeader set INFO_DOCUMENT_ROOT "%{INFO_DOCUMENT_ROOT}e"
RequestHeader set INFO_GATEWAY_INTERFACE "%{INFO_GATEWAY_INTERFACE}e"
RequestHeader set INFO_HTTPS "%{INFO_HTTPS}e"
RequestHeader set INFO_HTTP_ACCEPT "%{INFO_HTTP_ACCEPT}e"
RequestHeader set INFO_HTTP_ACCEPT_CHARSET "%{INFO_HTTP_ACCEPT_CHARSET}e"
RequestHeader set INFO_HTTP_ACCEPT_ENCODING "%{INFO_HTTP_ACCEPT_ENCODING}e"
RequestHeader set INFO_HTTP_ACCEPT_LANGUAGE "%{INFO_HTTP_ACCEPT_LANGUAGE}e"
RequestHeader set INFO_HTTP_CACHE_CONTROL "%{INFO_HTTP_CACHE_CONTROL}e"
RequestHeader set INFO_HTTP_CONNECTION "%{INFO_HTTP_CONNECTION}e"
RequestHeader set INFO_HTTP_COOKIE "%{INFO_HTTP_COOKIE}e"
RequestHeader set INFO_HTTP_FORWARDED "%{INFO_HTTP_FORWARDED}e"
RequestHeader set INFO_HTTP_HOST "%{INFO_HTTP_HOST}e"
RequestHeader set INFO_HTTP_KEEP_ALIVE "%{INFO_HTTP_KEEP_ALIVE}e"
RequestHeader set INFO_HTTP_MOD_SECURITY_MESSAGE "%{INFO_HTTP_MOD_SECURITY_MESSAGE}e"
RequestHeader set INFO_HTTP_PROXY_CONNECTION "%{INFO_HTTP_PROXY_CONNECTION}e"
RequestHeader set INFO_HTTP_REFERER "%{INFO_HTTP_REFERER}e"
RequestHeader set INFO_HTTP_USER_AGENT "%{INFO_HTTP_USER_AGENT}e"
RequestHeader set INFO_IS_SUBREQ "%{INFO_IS_SUBREQ}e"
RequestHeader set INFO_ORIG_PATH_INFO "%{INFO_ORIG_PATH_INFO}e"
RequestHeader set INFO_ORIG_PATH_TRANSLATED "%{INFO_ORIG_PATH_TRANSLATED}e"
RequestHeader set INFO_ORIG_SCRIPT_FILENAME "%{INFO_ORIG_SCRIPT_FILENAME}e"
RequestHeader set INFO_ORIG_SCRIPT_NAME "%{INFO_ORIG_SCRIPT_NAME}e"
RequestHeader set INFO_PATH "%{INFO_PATH}e"
RequestHeader set INFO_PATH_INFO "%{INFO_PATH_INFO}e"
RequestHeader set INFO_PHP_SELF "%{INFO_PHP_SELF}e"
RequestHeader set INFO_QUERY_STRING "%{INFO_QUERY_STRING}e"
RequestHeader set INFO_REDIRECT_QUERY_STRING "%{INFO_REDIRECT_QUERY_STRING}e"
RequestHeader set INFO_REDIRECT_REMOTE_USER "%{INFO_REDIRECT_REMOTE_USER}e"
RequestHeader set INFO_REDIRECT_STATUS "%{INFO_REDIRECT_STATUS}e"
RequestHeader set INFO_REDIRECT_URL "%{INFO_REDIRECT_URL}e"
RequestHeader set INFO_REMOTE_ADDR "%{INFO_REMOTE_ADDR}e"
RequestHeader set INFO_REMOTE_HOST "%{INFO_REMOTE_HOST}e"
RequestHeader set INFO_REMOTE_IDENT "%{INFO_REMOTE_IDENT}e"
RequestHeader set INFO_REMOTE_PORT "%{INFO_REMOTE_PORT}e"
RequestHeader set INFO_REMOTE_USER "%{INFO_REMOTE_USER}e"
RequestHeader set INFO_REQUEST_FILENAME "%{INFO_REQUEST_FILENAME}e"
RequestHeader set INFO_REQUEST_METHOD "%{INFO_REQUEST_METHOD}e"
RequestHeader set INFO_REQUEST_TIME "%{INFO_REQUEST_TIME}e"
RequestHeader set INFO_REQUEST_URI "%{INFO_REQUEST_URI}e"
RequestHeader set INFO_SCRIPT_FILENAME "%{INFO_SCRIPT_FILENAME}e"
RequestHeader set INFO_SCRIPT_GROUP "%{INFO_SCRIPT_GROUP}e"
RequestHeader set INFO_SCRIPT_NAME "%{INFO_SCRIPT_NAME}e"
RequestHeader set INFO_SCRIPT_URI "%{INFO_SCRIPT_URI}e"
RequestHeader set INFO_SCRIPT_URL "%{INFO_SCRIPT_URL}e"
RequestHeader set INFO_SCRIPT_USER "%{INFO_SCRIPT_USER}e"
RequestHeader set INFO_SERVER_ADDR "%{INFO_SERVER_ADDR}e"
RequestHeader set INFO_SERVER_ADMIN "%{INFO_SERVER_ADMIN}e"
RequestHeader set INFO_SERVER_NAME "%{INFO_SERVER_NAME}e"
RequestHeader set INFO_SERVER_PORT "%{INFO_SERVER_PORT}e"
RequestHeader set INFO_SERVER_PROTOCOL "%{INFO_SERVER_PROTOCOL}e"
RequestHeader set INFO_SERVER_SIGNATURE "%{INFO_SERVER_SIGNATURE}e"
RequestHeader set INFO_SERVER_SOFTWARE "%{INFO_SERVER_SOFTWARE}e"
RequestHeader set INFO_THE_REQUEST "%{INFO_THE_REQUEST}e"
RequestHeader set INFO_TIME "%{INFO_TIME}e"
RequestHeader set INFO_TIME_DAY "%{INFO_TIME_DAY}e"
RequestHeader set INFO_TIME_HOUR "%{INFO_TIME_HOUR}e"
RequestHeader set INFO_TIME_MIN "%{INFO_TIME_MIN}e"
RequestHeader set INFO_TIME_MON "%{INFO_TIME_MON}e"
RequestHeader set INFO_TIME_SEC "%{INFO_TIME_SEC}e"
RequestHeader set INFO_TIME_WDAY "%{INFO_TIME_WDAY}e"
RequestHeader set INFO_TIME_YEAR "%{INFO_TIME_YEAR}e"
RequestHeader set INFO_TZ "%{INFO_TZ}e"
RequestHeader set INFO_UNIQUE_ID "%{INFO_UNIQUE_ID}e"

Mod_Rewrite Variables Decoded!

[API_VERSION] => 20020903:12
[AUTH_TYPE] => Digest
[DOCUMENT_ROOT] => /home/user/www_root/askapache.com
[HTTPS] => off
[HTTP_ACCEPT] => text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
[HTTP_COOKIE] => PHPSESSID=752ee6d56e15f305233e30045987e5ce568c034; __qca=1176541225-59967328-5223185;
[HTTP_HOST] => www.askapache.com
[HTTP_REFERER] => http://www.askapache.com/protest/index.php?askapache=awesomeness&you=rock
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
[IS_SUBREQ] => false
[QUERY_STRING] => e=404
[REMOTE_ADDR] => 22.162.144.211
[REMOTE_HOST] => 22.162.144.211
[REMOTE_PORT] => 4511
[REMOTE_USER] => administrator
[REQUEST_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[REQUEST_METHOD] => GET
[REQUEST_URI] => /protest/index.php
[SCRIPT_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[SCRIPT_GROUP] => daemonu
[SCRIPT_USER] => askapache
[SERVER_ADDR] => 208.113.134.190
[SERVER_ADMIN] => webmaster@askapache.com
[SERVER_NAME] => www.askapache.com
[SERVER_PORT] => 80
[SERVER_PROTOCOL] => HTTP/1.1
[SERVER_SOFTWARE] => Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2
[THE_REQUEST] => GET /protest/adf HTTP/1.1
[TIME] => 20080820014309
[TIME_DAY] => 20
[TIME_HOUR] => 01
[TIME_MIN] => 43
[TIME_MON] => 08
[TIME_SEC] => 09
[TIME_WDAY] => 3
[TIME_YEAR] => 2008

Request using HTTPS

[API_VERSION] => 20020903:12
[AUTH_TYPE] => Digest
[DOCUMENT_ROOT] => /home/user/www_root/askapache.com
[HTTPS] => on
[HTTP_ACCEPT] => text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
[HTTP_COOKIE] => PHPSESSID=752ee6d56e15f305233e30045987e5ce568c034; __qca=1176541225-59967328-5223185;
[HTTP_HOST] => www.askapache.com
[HTTP_REFERER] => http://www.askapache.com/protest/index.php?askapache=awesomeness&you=rock
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16
[IS_SUBREQ] => false
[QUERY_STRING] => hi=you&whats=&you
[REMOTE_ADDR] => 22.162.144.211
[REMOTE_HOST] => 22.162.144.211
[REMOTE_PORT] => 4605
[REMOTE_USER] => administrator
[REQUEST_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[REQUEST_METHOD] => GET
[REQUEST_URI] => /protest/index.php
[SCRIPT_FILENAME] => /home/user/www_root/askapache.com/protest/index.php
[SCRIPT_GROUP] => daemonu
[SCRIPT_USER] => askapache
[SERVER_ADDR] => 208.113.134.190
[SERVER_ADMIN] => webmaster@askapache.com
[SERVER_NAME] => www.askapache.com
[SERVER_PORT] => 443
[SERVER_PROTOCOL] => HTTP/1.1
[SERVER_SOFTWARE] => Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.61 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2
[THE_REQUEST] => GET /protest/index.php?hi=you&whats=&you HTTP/1.1
[TIME] => 20080820015016
[TIME_DAY] => 20
[TIME_HOUR] => 01
[TIME_MIN] => 50
[TIME_MON] => 08
[TIME_SEC] => 16
[TIME_WDAY] => 3
[TIME_YEAR] => 2008

Emulating ErrorDocuments with Mod_Rewrite

The ErrorDocument directive is helpful because an errordocument is called differently then a normal file, and it contains special variables to help an admin debug.

I’ve wanted to use a RewriteCond + a RewriteRule to cause an Apache ErrorDocument to be displayed for a long time… I finally figured it out. Simply use the HTTP STATUS CODE trick in combination with a simple RewriteRule to trigger an Apache ErrorDocument.

This code emulates the internal 404 process Apache goes through.. If the file is not found it requests the /test/trigger-error/404 internally which triggers the 404 ErrorDocument.

ErrorDocument 404 /test/errordocument/404.html
Redirect 404 /test/trigger-error/404
 
RewriteEngine On
RewriteBase /
RewriteCond %{ENV:REDIRECT_STATUS} !=404
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* /test/trigger-error/404 [L]

Big Deal.. you might say… well consider that this works with any status code, and using this method you now have the power to trigger any errordocument page based on any kind of rewritecond. I’ll be writing about some practical uses for this powerful method in the coming weeks, but heres a good example now so you can see how it can be used.

This bit of code Triggers the 505 HTTP Version Not Supported When a request is made to the server with a protocol other than 1.1.

ErrorDocument 505 /test/errordocument/505.html
Redirect 505 /test/trigger-error/505
 
RewriteEngine On
RewriteBase /
 
RewriteCond %{ENV:REDIRECT_STATUS} !=505
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /.*\ HTTP/(0\.9|1\.0|1\.1) [NC]
RewriteRule .* /test/trigger-error/505 [L]

YES! I realize I didn’t explain that very well, I didn’t realize it was that complicated.. I wanted to go into how to use these advanced tricks and methods to achieve some really cool stuff, but explaining just this little bit took me awhile and I’m out of page space!

So play around with this and I’ll post back some of the untapped sicknesses you can give a website with such powerful methods at your disposal.

Ralf S. Engelschall

/*
 *  URL Rewriting Module
 *
 *  This module uses a rule-based rewriting engine (based on a
 *  regular-expression parser) to rewrite requested URLs on the fly.
 *
 *  It supports an unlimited number of additional rule conditions (which can
 *  operate on a lot of variables, even on HTTP headers) for granular
 *  matching and even external database lookups (either via plain text
 *  tables, DBM hash files or even external processes) for advanced URL
 *  substitution.
 *
 *  It operates on the full URLs (including the PATH_INFO part) both in
 *  per-server context (httpd.conf) and per-dir context (.htaccess) and even
 *  can generate QUERY_STRING parts on result.   The rewriting result finally
 *  can lead to internal subprocessing, external request redirection or even
 *  to internal proxy throughput.
 *
 *  This module was originally written in April 1996 and
 *  gifted exclusively to the The Apache Software Foundation in July 1997 by
 *
 *      Ralf S. Engelschall
 *      rse engelschall.com
 *      www.engelschall.com
 */

Crazy Advanced Mod_Rewrite Tutorial originally appeared on AskApache.

Raw Speed Benefit

This eliminates millions of bandwidth and resource robbing 304 If-Modified-Since requests.

Renaming links vs. Renaming files

On the server my files are named apache.css and apache.js, but in the xhtml I point to them using the names apache-113.css and apache-113.js, after I change the file I just add 1 to the number, and the new file is cached. They are internally redirected to apache.css and apache.js (invisible to the user) The concept is similar to a “shortcut” in windows or a symlink in BSD. The trick is that I never actually rename the files on the server. I just rename them in the html. That means apache-135.css is served from the file apache.css but the browser/cache only see and know about apache-135.css.

XHTML

NOTE: You can do your own investigating of this sites source code and HTTP headers to see this whole system in action

<link href="http://z.askapache.com/z/c/apache-0031.css" rel="stylesheet" type="text/css" />
<script src="http://z.askapache.com/z/j/apache-0031.js" type="text/javascript"></script>

mod_rewrite code for htaccess or httpd.conf

Updated: 10/20/2008

RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ /z/([a-z]+)/(.+)-([0-9]+)\.(js|css).*\ HTTP/ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .+ /z/%2/%3.%5 [NC,L]

Alternate mod_rewrite code

RewriteEngine On
RewriteBase /
RewriteRule ^([cij]+)(/?[a-z]*)/([a-z]+)-([0-9]+)\.([a-z]+)$ /$1$2/$3.$5 [L]

Ideal Caching Scheme

Ok so you want the xhtml to be the only file that isn’t cached without being validated, its simple to setup your own caching scheme.

Bad Cache information for a file with the ?v=foo hack

This object will be fresh for 1 week. It can be validated with Last-Modified. This object requests that a Cookie be set; this makes it and other pages affected automatically stale; clients must check them upon every request. Because it contains a query (‘?’) in the URL, many caches will not keep this object.

mod_rewrite Fix for Caching Updated Files originally appeared on AskApache.

I’ve used just about every CMS/Blog/Forum/Trac/Gallery/etc) and really didn’t like a lot of the way they coded… I could use php but I didn’t KNOW php.. so I’ve had to learn php also, and it was tough to learn the advanced class usage and all the other language specific (but similar) constructs for php. It was especially difficult (but fun and challenging) to program so as to be compatible with php4 and php5 (Such is WordPress). But I kept at it, and soon you can decide for yourself what to make of it.

I can code in plenty of languages (bash, lua, windows .bat and vbs, ocaml, big fan of all things shell) and can work my way through C and even sorta somewhat with assembly. Assembly is the hardest, by far, I’m into easy and powerful languages like Python, Javascript, perl, php, ruby, and CGI. I’ve used PHP for a long time to do various things, but never to build software projects like this. Once I noticed WordPress’s core .php files and the excellent programming I wanted to try and learn hot to do it. The WordPress code is some of the best I’ve seen. It appears the way they built it was planned, and not just dreamt up while typing that I can’t help but do. Every time I read through the core code I learn a new trick or very nice way to do something. Those guys are really good, and I think WordPress is going to dominate for a long long time.

The Strategy

The Password Protection (passpro) plugin has a lot of complex stuff going on, especially for a newbie to PHP and WordPress like me, so after refactoring the whole thing at least 5 times I decided to modify my approach, and wrote the AskApache Google 404 Plugin as a way to practice on a simpler piece of code, while at the same time providing a plugin of value. Eventually I stopped thinking I could just code the whole thing in one sit-down with a stream-of-consciousness, and had to instead modularize the code and focus in on each part before moving to the next (I go without a plan because its fun, just not the most productive, but again, I’m not a programmer in the scientific sense.).

So I decided I had to really learn how WordPress Plugins work, filters, hooks, actions, and basically comfortability at reverse-engineering code, (Im a beginner for the last time), and so with the upcoming release of the AskApache Google 404 Plugin I have succeeded in making an incredibly stable plugin. That way I only have to worry about what the aapasspro plugin is doing, instead of trying to fit it into a framework.

AskApache Google 404 Upgrade

I think its rather unusual to develop a nice plugin like this 404 handler merely for the purpose of improving upon another plugin, but hey it worked. As of 08/03/2009 14:06PM EST I have about 1 hour left of finishing touches to release this upgrade. But as you cantell by my badly edited posts, I don’t have a lot of time to myself. An hour here and there is about it. So it could be up to 2 weeks before I actually have the time to commit the release to the repo. On a sidenote, have you checked out Windows 7 News? I’ve been contracted to do some technical work for them and thought they had an excellent site.

But keep in mind, the 404 PLugin is just where I practice for the passpro plugin, which truly does have features that no other software like it has ever had. I understand the technology behind this plugin, and know it would really have a great impact on improving the Web (esp. WordPress) for all of us, I’ve just had to learn how to make it.

AskApache Password Protection

Probably still a couple weeks away, this plugin is the ultimate culmination of apache hackers dreams, at least those on shared servers (who may be interested in learning how to bypass security of said servers).. So this is something I have much too fun with doing what I like to do.. network/protocol-level security. I’ve examined the source code for many software packages that I use or have used to audit a server’s security, and this simple php plugin in most instances can enumerate with accuraccy most of the server’s setup in about a minute. The catch (and the file permission problems I had to find a workaround too) is that this software is launched on the server, not remotely against the server.

Some of the software I examined was whiskers, nessus, nmap, hping, mozilla source, wireshark, ncftp, netcat, etc.. The closest comparison to the socket-level class I’ve hacked together to those is wireshark. Except that wireshark only interprets (captures) the data passing over the wire, while this class does that and in fact sends and receives the data like netcat or nmap. Its really more similar to metasploit, and can easily be used to send hex, binary, ascii, or any type of payload to the remote or local host.

The Upgrades Begin

Well I started working on them a long time ago. Both the Password Protection plugin and the Google 404 plugin needed serious work. And I finally have it all figured out. Essentially I would work on one and finish an upgrade, but I just wasn’t happy with it and I wold start all over again, refactoring the code. So as I put the finishing touches on those 2 plugins keep an eye out. They are major upgrades. I was able to meet all the goals I had for them, and came up with a lot of more improvements during the process.One of the main things I needed was a socket-level class to perform all kinds of checks and tests on. I need this also for my crazy cache plugin, which my blog is currently using , and I have a 2 more really nice pplugins I use that also needed access to a network class. I wrote about what I was doing with fsockopen, and I’ve been improving on that example ever since. I use this class to do some really powerful and exciting stuff, but you’ll see it soon enough. As an indication of ‘getting it right’ for the Password Protection plugin, the plugin will now work on Windows, Apache, IIS, Lighthttpd, and will even work running on a blackberry web server. So now everyone using wordpress can at least get some security()

Many of the the other improvements focus on using the fsockopen class and .htaccess tricks to basically enumerate and discover all the different capabilities of your particular server; That way you can learn about all the features and security that are possible for your specific server, and the securty modules wi8ll be geared for that as well. FINALLY this plugin is going to be stable, and I just cant wait to see how people react when they learn all great capability their Apache-based Server has that they didn’t have a clue about. Its amazing in that sense, and hackers will love theh way it works.. but your server admins will love it even more because its entirely 100% focused on helping you to set your site up (if you have Apache) to keep spammers out, to keep virii-serving robots and their log-hogging exploit requests and CPU/Mem robiing 404 errors off of your servers for real. This will have a noticeable affect to whoever is running the server. As you can tell.. I am pumped!

Apache can only be influenced by the main server configs and by .htaccess files. Not by php, not by perl, and the main configs are almost never accessible to the masses. But .htaccess files are. And many hosting providers allow and enable .htaccess files, a configuration file for your web server. The advanced features and capabilities of Apache were out of reach for most of us, it just wasn’t possible to enumerate or access, and most hosting providers are infamous for their lack of .htaccess (customer) support. This plugin goes around those problems to give the power back to the people.

y creating custom .htaccess files containing unpublished .htaccess tricks and techniques and combining that with the use of socket-level networking from WordPress (PHP) using fsockopen, we can effectively enumerate and discover an incredible amount of features and settings you will be able to control and use with this plugin.

Here are a few examples of the capabilities of this plugin, some of which I believe no other software can do.. (Open source free to copy!).

1. Current Version of Apache (Down to the API Version)
2. List of ALL Modules currently enabled by Apache (Such as Mod_Rewrite)
3. List of ALL Directives enabled by EACH enabled Module.
4. Enumerate .htaccess Overrides, Context Permissions
5. Test for any builtin Handlers (like the status handler screenshot)
6. Configure SSI (http://www.askapache.com/htaccess/advanced-htaccess-ssi.html#htaccess-ssi-security)

March 1, 2009
I would focus on the method that WordPress uses. The code they have now (2.8 bleeding-edge) still isn’t where it needs to be, but this is some difficult stuff and they have a brilliant start, it’ll work.. just a question of when.

The main issue with the password protection plugin working for some people and not others is due to file permission configurations. The plugin attempts to write/modify files in your blog’s root directory.

November 05, 2008
To make a long story short, I downloaded each major release of the apache httpd source code starting at version 1.3.0 and finishing with version 2.2.11, I then compiled each version and built a HTTPD from source for all these apache versions.

• 1.3.0
• 1.3.1
• 1.3.11
• 1.3.12
• 1.3.14
• 1.3.17
• 1.3.19
• 1.3.2
• 1.3.20
• 1.3.22
• 1.3.23
• 1.3.24
• 1.3.27
• 1.3.28

• 1.3.29
• 1.3.3
• 1.3.31
• 1.3.32
• 1.3.33
• 1.3.34
• 1.3.35
• 1.3.36
• 1.3.37
• 1.3.39
• 1.3.4
• 1.3.41
• 1.3.6
• 1.3.9

• 2.0.35
• 2.0.36
• 2.0.39
• 2.0.40
• 2.0.42
• 2.0.43
• 2.0.44
• 2.0.45
• 2.0.46
• 2.0.47
• 2.0.48
• 2.0.49
• 2.0.50
• 2.0.51

• 2.0.52
• 2.0.53
• 2.0.54
• 2.0.55
• 2.0.58
• 2.0.59
• 2.0.61
• 2.0.63
• 2.1.3-beta
• 2.1.6-alpha
• 2.1.7-beta
• 2.1.8-beta
• 2.1.9-beta

• 2.2.0
• 2.2.10
• 2.2.2
• 2.2.3
• 2.2.4
• 2.2.6
• 2.2.8
• 2.2.9
• 2.2.10
• 2.2.11

Then I went through each version and determined the compatible modules for that version, and I’m pretty confident that I was also able to find each and every directive allowed by the compatible modules for that version (including core directives). See .htaccess directive list. Basically I can now test a server using a variety of methods and determine almost 100% accurately what version of Apache (down to the API) is running, what modules (and versions) are enabled, and each and every directive that is allowed or disallowed for that version. So this is so awesome because now we can enable all sorts of additional security features.

Htaccess enabled Modules

Here are most of the modules that come with Apache. Each one can have new commands that can be used in .htaccess file scopes.

mod_actions, mod_alias, mod_asis, mod_auth_basic, mod_auth_digest, mod_authn_anon, mod_authn_dbd, mod_authn_dbm, mod_authn_default, mod_authn_file, mod_authz_dbm, mod_authz_default, mod_authz_groupfile, mod_authz_host, mod_authz_owner, mod_authz_user, mod_autoindex, mod_cache, mod_cern_meta, mod_cgi, mod_dav, mod_dav_fs, mod_dbd, mod_deflate, mod_dir, mod_disk_cache, mod_dumpio, mod_env, mod_expires, mod_ext_filter, mod_file_cache, mod_filter, mod_headers, mod_ident, mod_imagemap, mod_include, mod_info, mod_log_config, mod_log_forensic, mod_logio, mod_mem_cache, mod_mime, mod_mime_magic, mod_negotiation, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_ftp, mod_proxy_http, mod_rewrite, mod_setenvif, mod_speling, mod_ssl, mod_status, mod_substitute, mod_unique_id, mod_userdir, mod_usertrack, mod_version, mod_vhost_alias

Debugging HTTP protocol

Check this out! I’m particularly happy about this feature, which outputs an exact trace of any requests made by the plugin (such as during the testing phase) by saving the actual raw data sent out on the wire using fsockopen, RX and TX. This is useful for a number of reasons, viewing your headers, finding Redirect Loops, testing RewriteRules, and following the request hop-by-hop for debugging. The below example shows 2 requests for 2 URIs. The first URI is protected using Digest Authentication, the 2nd shows Basic.

 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Digest realm="do or die", nonce="03328f3ec7c7b", algorithm=MD5, domain="/", qop="auth"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 882
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Digest username="test",realm="do or die",nonce="03328f3ec7c7b",uri="/htaccess/index.txt?testing=query",
cnonce="82d057852a9dc497",nc=00000001,algorithm=MD5,response="9d476e9ea3",qop="auth"
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Authentication-Info: rspauth="9051b01ee26dd62b3e2b40dada694f45", cnonce="82d057852a9dc497", nc=00000001, qop=auth
Last-Modified: Tue, 21 Jul 2009 23:56:00 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````
 
 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Basic realm="Po Pimping"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 878
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Basic adfAGAltcA==
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 05:54:39 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````

.htaccess Directives

AcceptFilter, AcceptMutex, AcceptPathInfo, AccessFileName, Action, AddAlt, AddAltByEncoding, AddAltByType, AddCharset, AddDefaultCharset, AddDescription, AddEncoding, AddHandler, AddIcon, AddIconByEncoding, AddIconByType, AddInputFilter, AddLanguage, AddModuleInfo, AddOutputFilter, AddOutputFilterByType, AddType, Alias, AliasMatch, AllowCONNECT, AllowEncodedSlashes, AllowOverride, Anonymous, Anonymous_Authoritative, Anonymous_LogEmail, Anonymous_MustGiveEmail, Anonymous_NoUserID, Anonymous_NoUserId, Anonymous_VerifyEmail, AssignUserId, AuthAuthoritative, AuthBasicAuthoritative, AuthBasicProvider, AuthDBDUserPWQuery, AuthDBDUserRealmQuery, AuthDBM, AuthDBMAuthoritative, AuthDBMGroupFile, AuthDBMType, AuthDBMUserFile, AuthDefaultAuthoritative, AuthDigestAlgorithm, AuthDigestDomain, AuthDigestFile, AuthDigestGroupFile, AuthDigestNcCheck, AuthDigestNonceFormat, AuthDigestNonceLifetime, AuthDigestProvider, AuthDigestQop, AuthDigestShmemSize, AuthGroupFile, AuthLDAPAuthzEnabled, AuthLDAPBindDN, AuthLDAPBindON, AuthLDAPBindPassword, AuthLDAPCharsetConfig, AuthLDAPCompareDNOnServer, AuthLDAPDereferenceAliases, AuthLDAPEnabled, AuthLDAPFrontPageHack, AuthLDAPGroupAttribute, AuthLDAPGroupAttributeIsDN, AuthLDAPRemoteUserAttribute, AuthLDAPRemoteUserIsDN, AuthLDAPStartTLS, AuthLDAPURL, AuthLDAPUrl, AuthName, AuthType, AuthUserFile, AuthzDBMAuthoritative, AuthzDBMType, AuthzDefaultAuthoritative, AuthzGroupFileAuthoritative, AuthzLDAPAuthoritative, AuthzOwnerAuthoritative, AuthzUserAuthoritative, BS2000Account, BalancerMember, BrowserMatch, BrowserMatchNoCase, BufferedLogs, CGIMapExtension, CacheDefaultExpire, CacheDirLength, CacheDirLevels, CacheDisable, CacheEnable, CacheExpiryCheck, CacheFile, CacheForceCompletion, CacheGcClean, CacheGcDaily, CacheGcInterval, CacheGcMemUsage, CacheGcUnused, CacheIgnoreCacheControl, CacheIgnoreHeaders, CacheIgnoreNoLastMod, CacheLastModifiedFactor, CacheMaxExpire, CacheMaxFileSize, CacheMaxStreamingBuffer, CacheMinFileSize, CacheNegotiatedDocs, CacheRoot, CacheSize, CacheStoreNoStore, CacheStorePrivate, CacheTimeMargin, CharsetDefault, CharsetOptions, CharsetSourceEnc, CheckCaseOnly, CheckSpelling, ChildPerUserId, ContentDigest, CookieDomain, CookieExpires, CookieLog, CookieName, CookieStyle, CookieTracking, CoreDumpDirectory, CustomLog, DAV, DAVDepthInfinity, DAVGenericLockDB, DAVMinTimeout, DBDExptime, DBDKeep, DBDMax, DBDMin, DBDParams, DBDPersist, DBDPrepareSQL, DBDriver, Dav, DavDepthInfinity, DavGenericLockDB, DavLockDB, DavMinTimeout, DefaultIcon, DefaultLanguage, DefaultType, DeflateBufferSize, DeflateCompressionLevel, DeflateFilterNote, DeflateMemLevel, DeflateWindowSize, Directory, DirectoryIndex, DirectoryMatch, DirectorySlash, DocumentRoot, DumpIOInput, DumpIOOutput, EnableExceptionHook, EnableMMAP, EnableSendfile, ErrorDocument, ErrorLog, Example, ExpiresActive, ExpiresByType, ExpiresDefault, ExtFilterDefine, ExtFilterOptions, ExtendedStatus, FancyIndexing, FileETag, Files, FilesMatch, FilterChain, FilterDeclare, FilterProtocol, FilterProvider, FilterTrace, ForceLanguagePriority, ForceType, ForensicLog, GprofDir, GracefulShutdownTimeout, Group, Header, HeaderName, HostNameLookups, HostnameLookups, ISAIPFakeAsync, ISAPIAppendLogToErrors, ISAPIAppendLogToQuery, ISAPICacheFile, ISAPIFakeAsync, ISAPILogNotSupported, ISAPIReadAheadBuffer, IdentityCheck, IdentityCheckTimeout, IfDefine, IfModule, IfVersion, ImapBase, ImapDefault, ImapMenu, Include, IndexIgnore, IndexOptions, IndexOrderDefault, IndexStyleSheet, KeepAlive, KeepAliveTimeout, LDAPCacheEntries, LDAPCacheTTL, LDAPCertDBPath, LDAPConnectionTimeout, LDAPOpCacheEntries, LDAPOpCacheTTL, LDAPSharedCacheFile, LDAPSharedCacheSize, LDAPTrustedClientCert, LDAPTrustedGlobalCert, LDAPTrustedMode, LDAPVerifyServerCert, LanguagePriority, Limit, LimitExcept, LimitInternalRecursion, LimitRequestBody, LimitRequestFields, LimitRequestFieldsize, LimitRequestLine, LimitXMLRequestBody, Listen, ListenBacklog, LoadFile, LoadModule, Location, LocationMatch, LockFile, LogFormat, LogLevel, MCacheMaxObjectCount, MCacheMaxObjectSize, MCacheMaxStreamingBuffer, MCacheMinObjectSize, MCacheRemovalAlgorithm, MCacheSize, MMapFile, MaxClients, MaxKeepAliveRequests, MaxMemFree, MaxRequestsPerChild, MaxSpareServers, MaxSpareThreads, MaxSpareThreadsPerChild, MaxThreads, MetaDir, MetaFiles, MetaSuffix, MimeMagicFile, MinSpareServers, MinSpareThreads, ModMimeUsePathInfo, MultiviewsMatch, NWSSLTrustedCerts, NWSSLUpgradeable, NameVirtualHost, NoProxy, NumServers, Options, PassEnv, PerlAccessHandler, PerlAuthenHandler, PerlAuthzHandler, PerlChildExitHandler, PerlChildInitHandler, PerlCleanupHandler, PerlDispatchHandler, PerlFixupHandler, PerlFreshRestart, PerlHandler, PerlHeaderParserHandler, PerlInitHandler, PerlLogHandler, PerlModule, PerlPassEnv, PerlPostReadRequestHandler, PerlRequire, PerlRestartHandler, PerlSendHeader, PerlSetEnv, PerlSetVar, PerlSetupEnv, PerlTaintCheck, PerlTransHandler, PerlTypeHandler, PerlWarn, PidFile, Port, Protocol, ProtocolEcho, Proxy, ProxyBadHeader, ProxyBlock, ProxyDomain, ProxyErrorOverride, ProxyFtpDirCharset, ProxyIOBufferSize, ProxyMatch, ProxyMaxForwards, ProxyPass, ProxyPassInterpolateEnv, ProxyPassMatch, ProxyPassReverse, ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath, ProxyPreserveHost, ProxyReceiveBufferSize, ProxyRemote, ProxyRemoteMatch, ProxyRequests, ProxySet, ProxyStatus, ProxyTimeout, ProxyVia, RLimitCPU, RLimitMEM, RLimitNPROC, ReadmeName, Redirect, RedirectMatch, RedirectPermanent, RedirectTemp, RemoveCharset, RemoveEncoding, RemoveHandler, RemoveInputFilter, RemoveLanguage, RemoveOutputFilter, RemoveType, RequestHeader, Require, RewriteBase, RewriteCond, RewriteEngine, RewriteLock, RewriteLog, RewriteLogLevel, RewriteMap, RewriteOptions, RewriteRule, SSIAccessEnable, SSIEndTag, SSIErrorMsg, SSIStartTag, SSITimeFormat, SSIUndefinedEcho, SSLCACertificateFile, SSLCACertificatePath, SSLCADNRequestFile, SSLCADNRequestPath, SSLCARevocationFile, SSLCARevocationPath, SSLCertificateChainFile, SSLCertificateFile, SSLCertificateKeyFile, SSLCipherSuite, SSLCryptoDevice, SSLEngine, SSLHonorCipherOrder, SSLLog, SSLLogLevel, SSLMutex, SSLOptions, SSLPassPhraseDialog, SSLProtocol, SSLProxyCACertificateFile, SSLProxyCACertificatePath, SSLProxyCARevocationFile, SSLProxyCARevocationPath, SSLProxyCipherSuite, SSLProxyEngine, SSLProxyMachineCertificateFile, SSLProxyMachineCertificatePath, SSLProxyProtocol, SSLProxyVerify, SSLProxyVerifyDepth, SSLRandomSeed, SSLRequire, SSLRequireSSL, SSLSessionCache, SSLSessionCacheTimeout, SSLUserName, SSLVerifyClient, SSLVerifyDepth, Satisfy, ScoreBoardFile, Script, ScriptAlias, ScriptAliasMatch, ScriptInterpreterSource, ScriptLog, ScriptLogBuffer, ScriptLogLength, ScriptStock, SecureListen, SendBufferSize, ServerAdmin, ServerAlias, ServerLimit, ServerName, ServerPath, ServerRoot, ServerSignature, ServerTokens, SetEnv, SetEnvIf, SetEnvIfNoCase, SetHandler, SetInputFilter, SetOutputFilter, StartServers, StartThreads, Substitute, SuexecUserGroup, ThreadLimit, ThreadStackSize, ThreadsPerChild, TimeOut, Timeout, TraceEnable, TransferLog, TypeAuthDBMUserFile, TypesConfig, UnsetEnv, UseCanonicalName, UseCanonicalPhysicalPort, User, UserDir, VirtualDocumentRoot, VirtualDocumentRootIP, VirtualHost, VirtualScriptAlias, VirtualScriptAliasIP, Win32DisableAcceptEx, XBitHack, allow, deny, order, php_admin_flag, php_admin_value, php_flag, php_value

You can view the plugins home page, old, or view it on the wordpress.org site.

An AskApache Plugin Upgrade to Rule them All originally appeared on AskApache.

Round Robin Concept

A few months ago I was given a free hosting account on HostGator to evaluate and tempt me away from DreamHost to become a Gator. I get a lot of these types of offers from time to time from the smaller Web companies who read AskApache.. but when I noticed the SSH access was jailed and saw how restrictive the shell was I felt like I was on a windows box.. I want a shell, cpanel sucks. I compile and run everything from the shell so thats was a deal-breaker and I sorta forgot all about it.

The goal is to add the HostGator server to be an exact mirror of the z.askapache.com domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.

Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)

DNS – Domain Name System

The Domain Name System is often analogized as a “phone book” for the Internet because it translates human-friendly computer hostnames into IP addresses. For example, www.askapache.com translates to 208.113.134.190. Every request for a human-friendly hostname first needs to be translated to the IP before the server can be queried. Normally each hostname corresponds to exactly 1 IP address. But in a Round Robin setup, the idea is to have the hostname correspond to multiple IP addresses, which are different servers that house the exact same content, resulting in some hosts requesting files from one server, and another host requesting files from the other server, resulting in less CPU resources and bandwidth.

Here is an the zone for z.askapache.com Round Robin records:

Round Robin DNS

Round robin DNS is a technique of load distribution, load balancing, or fault-tolerance provisioning multiple, redundant Internet Protocol service hosts, e.g., Web servers, FTP servers, by managing the Domain Name System’s (DNS) responses to address requests from client computers according to an appropriate statistical model.

In its simplest implementation Round-robin DNS works by responding to DNS requests not only with a single IP address, but a list of IP addresses of several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query so that on different connection attempts clients would receive service from different providers, thus distributing the overall load among servers.

Round robin DNS is often used for balancing the load of geographically-distributed Web servers. For example, a company has one domain name and three identical web sites residing on three servers with three different IP addresses. When one user accesses the home page it will be sent to the first IP address. The second user who accesses the home page will be sent to the next IP address, and the third user will be sent to the third IP address. In each case, once the IP address is given out, it goes to the end of the list. The fourth user, therefore, will be sent to the first IP address, and so forth.

Although easy to implement, round robin DNS has problematic drawbacks, such as those arising from record caching in the DNS hierarchy itself, as well as client-side address caching and reuse, the combination of which can be difficult to manage. Round robin DNS should not solely be relied upon for service availability. If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service.

Does Round Robin Work

Definately. I can look at the access logs for both servers and see that they are splitting the requests nicely. It is definately not an exact split however, look at these statistics and you’ll see what I mean.

$ dig @ns1.dreamhost.com +authority +all z.askapache.com
 
;; ANSWER SECTION:
z.askapache.com.        14400   IN      A       69.56.174.114
z.askapache.com.        14400   IN      A       64.111.114.111
 
$ dig @ns1.dreamhost.com +authority +all z.askapache.com
 
;; ANSWER SECTION:
z.askapache.com.        14400   IN      A       64.111.114.111
z.askapache.com.        14400   IN      A       69.56.174.114

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used which has the effect of significantly increasing the load on the DNS (and is not always implemented consistently). In this case the cure may be worse than the disease Good news we have good load balancing on our web servers. Bad news we need 17 more DNS servers!. Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static list, loads should be reasonably balanced .

Full root server query

Tracing to z.askapache.com[a] via A.ROOT-SERVERS.NET, maximum of 3 retries
A.ROOT-SERVERS.NET [.] (198.41.0.4)
 |\___ L.GTLD-SERVERS.NET [com] (192.41.162.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ D.GTLD-SERVERS.NET [com] (192.31.80.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ J.GTLD-SERVERS.NET [com] (192.48.79.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ H.GTLD-SERVERS.NET [com] (192.54.112.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ E.GTLD-SERVERS.NET [com] (192.12.94.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ F.GTLD-SERVERS.NET [com] (192.35.51.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ I.GTLD-SERVERS.NET [com] (192.43.172.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ G.GTLD-SERVERS.NET [com] (192.42.93.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ B.GTLD-SERVERS.NET [com] (2001:0503:231d:0000:0000:0000:0002:0030) Not queried
 |\___ B.GTLD-SERVERS.NET [com] (192.33.14.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ A.GTLD-SERVERS.NET [com] (2001:0503:a83e:0000:0000:0000:0002:0030) Not queried
 |\___ A.GTLD-SERVERS.NET [com] (192.5.6.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ C.GTLD-SERVERS.NET [com] (192.26.92.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ M.GTLD-SERVERS.NET [com] (192.55.83.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
  \___ K.GTLD-SERVERS.NET [com] (192.52.178.30)
       |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
       |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
        \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       z.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       z.askapache.com -> 69.56.174.114

Live Online DNS Tools

• DNSstuff.com – Your Destination for DNS and Networking Tools
• ZoneCheck
• Quick Check – Pingability.com
• Squishywishywoo: complete dns traversal checking
• ZoneCut DNS
• DNS Sleuth
• DNS test – Domain name DNS test – pweb.cz
• OpenDNS – Support – CacheCheck
• DNSReport
• DNSTrace
• intoDNS
• DNSthru.com
• DNS Doctor
• Power Check

More Reading

• HOWTO – Configure Load Balancing
• Load Sharing with DNS

RFC’s related to DNS

• RFC 920: Specified original TLDs: .arpa, .com, .edu, .org, .gov, .mil and two-character country codes
• RFC 1032: Domain administrators guide
• RFC 1033: Domain administrators operations guide
• RFC 1034: Domain Names – Concepts and Facilities.
• RFC 1035: Domain Names – Implementation and Specification
• RFC 1101: DNS Encodings of Network Names and Other Types
• RFC 1123: Requirements for Internet Hosts — Application and Support
• RFC 1912: Common DNS Operational and Configuration Errors
• RFC 1995: Incremental Zone Transfer in DNS
• RFC 1996: A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)
• RFC 2136: Dynamic Updates in the domain name system (DNS UPDATE)
• RFC 2181: Clarifications to the DNS Specification
• RFC 2182: Selection and Operation of Secondary DNS Servers
• RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
• RFC 2317: Classless IN-ADDR.ARPA delegation
• RFC 2671: Extension Mechanisms for DNS (EDNS0)
• RFC 3597: Handling of Unknown DNS Resource Record (RR) Types
• RFC 3696: Application Techniques for Checking and Transformation of Names
• RFC 4343: Domain Name System (DNS) Case Insensitivity Clarification
• RFC 4592: The Role of Wildcards in the Domain Name System
• RFC 4892: Requirements for a Mechanism Identifying a Name Server Instance
• RFC 5001: DNS Name Server Identifier Option (NSID)
• RFC 5395: Domain Name System (DNS) IANA Considerations

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Table of Contents

• Round Robin Concept
• DNS – Domain Name System
• Round Robin DNS
• Efficacy of DNS Load Balancing
• Live Online DNS Tools
• More Reading
• RFC’s related to DNS

DNS Round Robin Configuration using Rsync over SSH originally appeared on AskApache.

For this article I will show how I setup a web mirror of z.askapache.com from my DreamHost server to my HostGator Server. For the transfer and synchronization of the directories we will be using rsync over SSH. We will also be automating this task using a cronjob with no user-interaction, so creating public and private keys for the ssh will be neccessary. Finally, I provide a simple shell script that prevents anyone from logging into your account with the created keys for any purpose other than to rsync.

Table of Contents

1. Rsync Synchronization Magic
2. Securing Rsync with SSH
3. Generate Keys with No Password
4. Custom SSH Connection Info
5. Creating Cronjob for Synchronization
6. Only Allow rsync
7. Rsync/SSH Debugging and Stats
8. Rsync Algorithm

Rsync Synchronization Magic

rsync

rsync is an open source file transfer program for Unix systems that uses the “rsync algorithm” which provides a very fast method for synchronizing files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

Securing Rsync with SSH

I NEVER transfer any unencrypted data around unless that data is transported encrypted and to a trusted recipient, (I use HTTPS for WordPress administration) and I haven’t had time to probe the HostGator system for security issues yet, so right away I decided I needed an automated way to securely transfer z.askapache.com files TO hostgator, while not allowing my hostgator account access back on my main server. So if the hostgator account were to get hacked somehow, the cracker/spammer wouldn’t have access back to my main server.

Generate Keys with No Password

First I created a private key, specifically a passwordless host key, meaning to gain access with ssh you only need to supply the key, not a password+key like normal.

[local@dreamhost] $ mkdir -p ~/.ssh && chmod 700 ~/.ssh
 
# Create the public and private keys
# public key at: z.askapache-hostgator.id_rsa.pub
# private key at: z.askapache-hostgator.id_rsa
[local@dreamhost] $ ssh-keygen -t rsa -b 2048 -f ~/.ssh/z.askapache-hostgator.id_rsa
 
# add the public key to remote hosts ~/.ssh/authorized_keys file
[local@dreamhost] $ ssh-copy-id -i ~/.ssh/z.askapache-hostgator.id_rsa.pub remoteuser@remotehost
 
# or use scp + ssh to add the public key
[local@dreamhost] $ scp ~/.ssh/z.askapache-hostgator.id_rsa.pub remoteuser@remotehost:/home/remoteuser/
[local@dreamhost] $ ssh remoteuser@remotehost
[gatoraskapache@gator] $ mkdir -p ~/.ssh && chmod 700 ~/.ssh
[gatoraskapache@gator] $ cat ~/z.askapache-hostgator.id_rsa.pub >> ~/.ssh/authorized_keys
[gatoraskapache@gator] $ chmod 600 ~/.ssh/authorized_keys

Custom SSH Connection Info

This helps alot, by adding this to the very top of my ~/.ssh/config file I don’t have to add all this stuff to the rsync command-line. Basically when I reference connecting to the host ‘gator’ it uses all these options. Very helpful and you can add as many entries as you want.

Host gator
   IdentityFile ~/.ssh/z.askapache-hostgator.id_rsa
   Port 2222
   Protocol 2
   User gatoraskapache
   HostName gator555.hostgator.com
   PasswordAuthentication no

Creating Cronjob for Synchronization

This cronjob runs every 30 minutes, copying all modified files from my local directory ~/z.askapache.com/ to the remote directory ~/public_html/z/. First I backup the current crontab, then I edit the crontab and add this.

crontab -l > .crontab-`date +%F.backup`; crontab -e

*/30 * * * * /usr/bin/rsync -e 'ssh' -rt --delete ~/z.askapache.com/ gator:'~/public_html/z/' 1>/dev/null
@midnight /usr/bin/find ~/z.askapache.com/ -type d ! -perm 755 -exec chmod 755 {} \; 1>/dev/null
@midnight /usr/bin/find ~/z.askapache.com/ -type f ! -perm 644 -exec chmod 644 {} \; 1>/dev/null

Those 2 find commands scheduled to run at midnight simply fix and permissions on files and directories in my static folder. They are all static files so there is no reason they need to have any other permission.

Only Allow rsync

This is SWEET! If you like candy that is.. It’s called each time anything logs into your machine using the password less key we created above, and it simple checks what command the login process is attempting to issue. To set this up you need to edit the ~/.ssh/authorized_keys file on the remote host and prefix your public key that you added with a command directive to execute a script:

command="/home/remoteuser/validate-rsync.sh" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwBhj6UCS7JbJ08C8pWJqCh2iXZMN7tXpYZh47f4gZZBwrNHZQ== localuser@dreamhost

validate-rsync.sh

For rsync requests it will always be rsync --server at the start of the command, so if the command is anything else then this script:

1. Sends you an email notifying you somethings up..
2. Moves the ~/.ssh folder to ~/.locked-ssh
3. Adds cronjob to move the folder back in about an hour.

#!/bin/bash
# Author: http://www.askapache.com
# Version: 1.2
# Date: 04-08-2009
 
# If the command used to login to ssh correctly starts with 'rsync --server'
# then exit this script and dont process the rest of the script
case "$SSH_ORIGINAL_COMMAND" in 'rsync --server'*) exit 0; ;; esac;
 
# the home directory where the .ssh folder is located
H=/home/remoteuser
 
# if there is a locked ssh folder, kill the rsync and die
[[ -d $H/.lssh ]] && echo "REJECTED" && exit 1
 
EMAIL=webmaster@askapache.com # notified about locked status
OC=$H/old-crontab.txt # the original crontab
NC=$H/new-crontab.txt #the new crontab
 
# When to unlock the rsync
UNLOCK_AT=$(( date +%M\ %k --date='30 minute 1 hour' ))
 
# move the .ssh to .lssh which locks all key-logins
mv $H/.ssh $H/.lssh
 
# mail a notice to the boss
echo $SSH_CONNECTION | mail -s 'RSYNC LOCKED!!!' "$EMAIL"
 
# subshell backs-up crontab then deletes active cron
(

 crontab -l > $OC &>/dev/null || echo -n #backup current crontab
 crontab -r >/dev/null 2>&1 || echo -n # delete current crontab
)
 
# subshell creates new crontab combined with old crontab
(
 # create new crontab
 echo -en "MAILTO='${EMAIL}'\n${UNLOCK_AT} * * * mv $H/.lssh $H/.ssh" >> $NC
 echo -n " && date|mail -s 'UNLOCKED!!!' '${EMAIL}' && crontab $OC || rm $OC && rm $OC" >> $NC

 # add old crontab to new crontab minus any MAILTO lines
 cat $OC | sed '/^MAILTO/d' >> $NC

 # load the new crontab and if it doesnt work notify boss
 crontab $NC || echo "manually mv .lssh to .ssh" mail -s 'CRONTAB PROBLEM!!!' "$EMAIL"

 # remove new crontab
 rm $NC
)
 
exit $?

Here is the cronjob entry it creates… This will enable the rsync again by moving the folder back, then it mails you to alert you that its back up, and finally the original crontab is restored.

MAILTO="askapache@gmail.com"
03 9 * * * mv ~/.locked-ssh ~/.ssh && date|mail -s 'RUNLOCKED!!!' "webmaster@askapache.com" && crontab ~/old-crontab.txt

Rsync/SSH Debugging and Stats

Adding the option -v to the ssh command, ie rsync -e 'ssh -vv' will give you a lot of debugging info.

By adding –stats to your rsync command you can get a detailed look at its efficacy.

Number of files: 14900
Number of files transferred: 0
Total file size: 1456832331 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 320551
File list generation time: 17.393 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 320571
Total bytes received: 20
 
sent 320571 bytes  received 20 bytes  17329.24 bytes/sec
total size is 1456832331  speedup is 4544.21

Rsync Algorithm

The rsync utility uses an algorithm (invented by the Australian computer programmer Andrew Tridgell) for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a different version of the same structure.

The recipient splits its copy of the file into fixed-size non-overlapping chunks, of size S, and computes two checksums for each chunk: the MD4 hash, and a weaker ‘rolling checksum’. It sends these checksums to the sender. Version 30 of the protocol (released with rsync version 3.0.0) now uses MD5 hashes rather than MD4.

The sender computes the rolling checksum for every chunk of size S in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of bytes n through n + S − 1 is R, the rolling checksum of bytes n + 1 through n + S can be computed from R, byte n, and byte n + S without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum, and from bytes 1 and 26.

The rolling checksum used in rsync is based on Mark Adler’s adler-32 checksum, which is used in zlib, and which itself is based on Fletcher’s checksum. The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the MD4 checksum for the matching block and by comparing it with the MD4 checksum sent by the recipient.

The sender then sends the recipient those parts of its file that did not match any of the recipient’s blocks, along with assembly instructions on how to merge these blocks into the recipient’s version. In practice, this creates a file identical to the sender’s copy. However, it is in principle possible that the recipient’s copy differs at this point from the sender’s: this can happen when the two files have different chunks that nonetheless possess the same MD4 hash and rolling checksum; the chances for this to happen are in practice extremely remote.

If the sender’s and recipient’s versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files.

While the rsync algorithm forms the heart of the rsync application that essentially optimizes transfers between two computers over TCP/IP, the rsync application supports other key features that aid significantly in data transfers or backup. They include compression and decompression of data block by block using zlib at sending and receiving ends, respectively, and support for protocols such as ssh that enables encrypted transmission of compressed and efficient differential data using rsync algorithm. Instead of ssh, stunnel can also be used to create an encrypted tunnel to secure the data transmitted.

Finally, rsync is capable of limiting the bandwidth consumed during a transfer, a useful feature that few other standard file transfer protocol offer.

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Mirroring an Entire Site using Rsync over SSH originally appeared on AskApache.

Welcome to the first generation of the new .htaccess tutorials/articles. Basically these are articles detailing my BEST (almost) successful .htaccess experiments, meaning this is the only place on the net you’ll find this technique. It’s home-grown .htaccess, and its some kush, good kush. Instead of just publishing my cryptic results and code in days past, whoah.. a working demo and the NSA building ;)

Using the Mod_Rewrite Variables Cheatsheet makes this example, and all advanced .htaccess code easier to understand. Well, advanced for me at least. I know alot of AskApache visitors are some of the top gurus in many IT fields, but mostly web developers or website owners like me.. This demo is pretty self-explanatory.. Try it out to see how simple of a task this .htaccess trick performs. And make sure you read the whole article as this htaccess technique can be used to do a heck of a lot more than this simple demo.

Set PDF Viewing Mode

Whoa pretty sweet huh? Bet you’ve never seen that before! As I explain the htaccess code that achieves this, keep in mind this is merely one simple application for this code. It’s much more advanced than your basic htaccess trick, notice how this htaccess acts like a php script, very unusual.. I really wanted to share this trick after I created it for one of my clients because this is the tip of the iceberg. Another use would be to display an alternate style sheet depending on a users theme preference. The coolest thing about this example IMHO is that it uses multiple advanced .htaccess ideas in order for it to work, most htaccess code on the net is very singular. This code uses mod_headers to set the Content-Disposition header for forcing a download and uses mod_rewrite to:

1. Send different Content-Type headers
2. Check the value of a cookie
3. Set environment variables for use later by mod_headers header directive

What’s Going On

There are 3 different ways for a server to send a pdf file in response to a request for one. This causes 3 different ways to open/view the pdf file in the clients browser.

1. The browser display’s a “Save File As” dialog, allowing you to save the file or open.
2. The browser opens the pdf file “Inline”, opening the pdf file in the browser like a web page.
3. The browser “Downloads” the pdf file automatically as an “Attachment” and then causes an external pdf reader program like adobe reader to open the file.

Some people prefer to have the option of saving the file to view later, some prefer opening it with an external program, and some just like the pdf file to load right in the browser… The point is that by using .htaccess, we can let them choose any of the 3 methods and save their preference for all further pdf files requested from our site by that user.

How It Works

When you click on one of the 3 demo buttons above, “Inline”, “Save As”, or “Download”, a cookie named askapache_pdf is saved in your browser using the javascript below, with the value being set to which button you clicked. Then when you request the pdf file the .htaccess code below uses mod_rewrite to read the value of the askapache_pdf cookie, and depending on which was your preference it will send alternate HTTP Headers that control how your browser handles the file.

Htaccess Demo File

For the demo I created the folder /storage/pdf/ and this is the .htaccess file at /storage/pdf/.htaccess

#
# The default Content-Type for .pdf files
# This will make .pdf files default Content-Type header have
# the value 'application/pdf' - but the default can be overridden by
# using RewriteRule with the [T='different/type']
#
AddType application/pdf .pdf
 
#
# Turn on the rewrite engine
# if its already on you dont need this
#
RewriteEngine On
 
#
# Skip RewriteRules if not .pdf request, like autoindexing
# The next [2] RewriteRule directives are specific for .pdf files
# so if the filename requested does not end in .pdf
# then the [S=2] instructs the next 2 RewriteRule
# directives to be completely skipped
#
RewriteRule !.*\.pdf$ - [S=2]
 
#
# The first RewriteCond checks to see if the askapache_pdf cookie
# is NOT set.  The second RewriteCond checks to see if the askapche_pdf
# cookie has the value of s, which is the value corresponding to
# someone clicking the "Save As" button.
#
# The [NC,OR] flag means that if the cookie askapache_pdf does not
# exist, OR (next cond) if the askapache_pdf cookie does exist and is set to 's'
# then process the RewriteRule.  If neither cond is true the rewriterule is skipped.
#
# If one of the RewriteCond is true, then the RewriteRule is processed.
# The RewriteRule applies to any/all requests (.*) but doesn't rewrite anything (-)
# This RewriteRule sets an Apache environment variable ASKAPACHE_PDFS to have the
# value of 1 if either rewritecond is true.  The variable can be checked by any directives
# following the rewriterule in the whole htaccess file.  The ASKAPACHE_PDFS ends in S
# because if this variable exists then it means the users preference is 'Save As'
#
# Notice that if the user requested the pdf file without selecting a preference
# i.e. no cookie exists, then the ASKAPACHE_PDFS variable is still set.
# This just lets us pick the default preference for them, in this example the
# default is 'Save As'
#
RewriteCond %{HTTP_COOKIE} !^.*askapache_pdf.*$ [NC,OR]
RewriteCond %{HTTP_COOKIE} ^.*askapache_pdf=s.*$ [NC]
RewriteRule .* - [E=ASKAPACHE_PDFS:1]
 
#
# The RewriteCond checks the askapache_pdf cookie for the value 'a'
# which 'a' represents 'Download'
#
# If the cookies value is 'a' then the RewriteRule overrides the default
# Content-Type from 'application/pdf' set with AddType earlier, to
# 'application/octet-stream', which is a special content-type that tells the browser
# that the file cannot be loaded by the browser 'Inline', but must be saved
# which will be opened by an external viewer depending on browser
#
RewriteCond %{HTTP_COOKIE} ^.*askapache_pdf=a.*$
RewriteRule .* - [T=application/octet-stream]
 
#
# This is superfly.  If the cookie/users-preference was 'Save As' (s)
# then the RewriteRule above the last one set the environment
# variable ASKAPACHE_PDFS to have the value 1.  The Header directive here
# is ONLY processed in that variable ASKAPACHE_PDFS exists.  That is what
# the end 'env=ASKAPACHE_PDFS' does, it is the condition that must be met or
# the Header directive is skipped.
#
# If the ASKAPACHE_PDFS environment variable set by RewriteRule does exist
# then the header directive adds the header 'Content-Disposition: attachment' to
# the normal Response Headers.  The 'Content-Disposition: attachment' header
# instructs your browser to present you with the 'Save As' dialog box
# allowing you to choose whether you want to save or open
#
Header set Content-Disposition "attachment" env=ASKAPACHE_PDFS

Unique HTTP Headers Returned

When it comes down to it, the following information is the 3 modes. Notice each one is different, because these headers are the only thing controlling how your browser handles the file.

Save As Mode (askapache_pdf=s)

Content-Disposition: attachment
Content-Type: application/pdf

Inline Mode (askapache_pdf=i)

Content-Type: application/pdf

Download Mode (askapache_pdf=a)

Content-Type: application/octet-stream

Javascript used by Demo

The best place for javascript is quirksmode, here is a definitive article on setting, reading, parsing, etc.. COOKIES.

  if(!gi('pdfr'))return;
  var pdfr=gi('pdfr');
  var cval=getCookie('askapache_pdf');

  if(cval=='i'){pdfr.innerHTML='Currently set to "Inline".';}
  else if(cval=='a'){pdfr.innerHTML='Currently set to "Download" mode.';}
  else if(cval=='s'){pdfr.innerHTML='Currently set to "Save As" mode.';}
 
  addMyEvent(gi('pdfi'),"mousedown",function(){setCookie("askapache_pdf", "i", "", "/", "www.askapache.com"); gi('pdfr').innerHTML = 'Changed mode to "Inline".'; return false; });
  addMyEvent(gi('pdfa'),"mousedown",function(){setCookie("askapache_pdf", "a", "", "/", "www.askapache.com"); gi('pdfr').innerHTML = 'Changed mode to "Download".'; return false; });
  addMyEvent(gi('pdfs'),"mousedown",function(){setCookie("askapache_pdf", "s", "", "/", "www.askapache.com"); gi('pdfr').innerHTML = 'Changed mode to "Save As".'; return false; });

Alternative Method – No Cookies + PHP

This is what I came up with first for my client, and then while programming the php I noticed.. Hey! I think I can do the same thing using .htaccess, which would save me on cpu/memory/potential security/etc.. but this works great too. Though you will need to hack the code to get it working probably..

Note that the .htaccess rewrite code I used here used FILENAME-i.pdf or FILENAME-s.pdf to pass the preference to the pdf-dl.php script, it also worked for FILENAME.pdf?i=i

pdf-dl.php

if (
!isset($_GET['file'])
|| ($f=$_GET['file'])===false
|| ($fp=@fopen($f,"rb"))===false
|| ($fi=pathinfo($f))===false
|| ($fi['fsize']=filesize($f))===false
|| strtolower($fi["extension"])!='pdf'
) die('Failed');

ob_start();
header('Accept-Ranges: bytes');
header("Content-Length: {$fi['fsize']}");
header('Content-Type: application/pdf');
if(!isset($_GET['i'])) header("Content-Disposition: attachment; filename=\"{$fi['basename']}\"");

$sent = 0;
while ( !feof($fp) && $sent < $fi['fsize'] && ($buf = fread($fp, 8192)) != '' ){
echo $buf;
$sent += strlen($buf);
flush();
ob_flush();
}
fclose($fp);
exit;
?>

Alternate Method .htaccess

#
# Deny direct request to pdf-dl.php file
#
RewriteCond %{THE_REQUEST} ^.*pdf-dl\.php.*$ [NC]
RewriteRule .* - [F]
 
#
# Handle PDF files named anything-i.pdf as inline
#
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ([^/]*)-i\.pdf$  /cgi-bin/pdf-dl.php?i=i&file=%{DOCUMENT_ROOT}/storage/pdf/$1.pdf [L,NC,QSA,S=1]
 
#
# Handle PDF files without -i.pdf as attachments
#
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ([^/]*)\.pdf$  /cgi-bin/pdf-dl.php?file=%{DOCUMENT_ROOT}/storage/pdf/$1.pdf [L,NC,QSA]

More Info

The following is more information about the Content-Dispositon header and related headers, and will make you an expert at this if you read it all.. (no thanks)

Interesting Reading

Here is the thread of the original draft proposal for the Content-Disposition header.

• Content-Disposition Header, Rens Troost - 22 Jun 1993
  • Re: Content-Disposition Header, Nathaniel Borenstein
    • Re: Content-Disposition Header, Gabe Beged-Dov
      • Re: Content-Disposition Header, Rens Troost
      • Re: Content-Disposition Header, Gabe Beged-Dov
      • Content-Disposition Header and multipart/alternative, Rens Troost
      • Re: Content-Disposition Header and multipart/alternative, Nathaniel Borenstein
    • Re: Content-Disposition Header, Keith Moore
      • Re: Content-Disposition Header, Nathaniel Borenstein
      • Re: Content-Disposition Header, Ed Levinson (Contractor)
      • Re: Content-Disposition Header, Keith Moore
      • Re: Content-Disposition Header, Rens Troost
      • Re: Content-Disposition Header, Harald Tveit Alvestrand
    • Re: Content-Disposition Header, Steve Dorner
      • Re: Content-Disposition Header, Rens Troost
        • Re: Content-Disposition Header, Keith Moore
    • Re: Content-Disposition Header, Carlyn M. Lowery

Intense Reading

• Using the MIME Headers Effectively
• Mail Content Disposition Values and Parameters
• Tags for the Identification of Languages
• The Internet Gopher Protocol (a distributed document search and retrieval protocol)
• Universal Resource Identifiers in WWW
• Uniform Resource Locators (URL)
• Hypertext Markup Language - 2.0
• Hypertext Transfer Protocol -- HTTP/1.0
• Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies
• Requirements for Internet Hosts -- Communication Layers
• Standard for The Format of ARPA Internet Text Messages
• Relative Uniform Resource Locators
• Standard for Interchange of USENET Messages
• Network News Transfer Protocol
• MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text
• Form-based File Upload in HTML
• Simple Mail Transfer Protocol
• Media Type Registration Procedure
• File Transfer Protocol
• Assigned Numbers
• Functional Requirements for Uniform Resource Names
• The Content-MD5 Header Field
• Renumbering Needs Work
• GZIP file format specification version 4.3
• Improving HTTP Latency
• Analysis of HTTP Performance
• Network Time Protocol (Version 3) Specification, Implementation and Analysis
• DEFLATE Compressed Data Format Specification version 1.3
• Analysis of HTTP Performance Problems,
• ZLIB Compressed Data Format Specification version 3.3
• An Extension to HTTP: Digest Access Authentication
• Hypertext Transfer Protocol -- HTTP/1.1
• Key words for use in RFCs to Indicate Requirement Levels
• Communicating Presentation Information in Internet Messages: The Content-Disposition Header
• Use and Interpretation of HTTP Version Numbers
• Common Internet Message Headers
• UTF-8, a transformation format of Unicode and ISO-10646
• Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types
• IETF Policy on Character Sets and Languages
• Uniform Resource Identifiers (URI): Generic Syntax and Semantics
• HTTP Authentication: Basic and Digest Access Authentication
• MIME E-mail Encapsulation of Aggregate Documents, such as HTML (MHTML)
• The Internet Standards Process -- Revision 3
• Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0)
• Multipurpose Internet Mail Extensions (MIME) Part Five: Conformance Criteria and Examples
• Communicating Presentation Information in Internet Messages: The Content-Disposition Header Field

Advanced Htaccess Demo/Example using Cookies, Headers, Rewrites originally appeared on AskApache.

Scale is based on the average worldwide traffic of htaccess in all years.

.htaccess vs. htaccess

htaccess vs. httpd.conf

mod_rewrite vs. mod_security

Web Server Comparisons

Of course IIS is well-known to someone like me… but who cares about the big servers when small, light, super-fast servers are on the rise? See this Comparison of Web Servers.

Apache HTTP vs. Windows IIS

Fnord, Nginx, LightHttpd

Fnord what is going on?

Web Programming Comparisons

An interesting look at the search engine trends of programming languages out there today…

Did Ruby Pass Perl!

Keep in mind that Google may be interpreting “ruby” to be a precious stone… and what is a “perl” exactly? ;)

Perl vs. PHP, Old Debate

SEO

Wow. Big surpise there ;)

Treo vs. Blackberry vs. Palm

I love my Blackberry Curve! I used to love my Sony Clie, and I won’t forget my many Palms!

AskApache WebSite

Google Trends also lets you compare websites, I can’t believe its been over a year for askapache.com, thanks for stopping by!

Google Trends – Rocks!

Fun, free, and helpful tool from who else?

Htaccess SEO Trends by Google originally appeared on AskApache.

1. Improved Directory Listings and SEO
2. Enhanced Error Pages
3. Dealing with Spam and Website Attacks

Improved Directory Listings and SEO

One way that I use SSI is for improved Directory Listing. Anyone here like SEO? Check out my WordPress uploads directory to get an idea. Basically I can customize the header and footer of each directory.. Pretty sweet, thanks Apache!

In your /uploads/.htaccess

# turn on auto-indexing and turn off SSI's ability to exec
Options None
Options SymLinksIfOwnerMatch Indexes IncludesNOEXEC
 
# we need to make sure files are displayed when requested, not executed or parsed
AddType text/plain .ini .sh .bsh .bash .csh .var .asc .md5 .sha .sha1 .cgi .pl .php .inc .asp .exe .bin
DefaultType text/plain
 
# turn on auto-indexing, with askapache-optimized options
IndexOptions FancyIndexing SuppressColumnSorting SuppressHTMLPreamble IconHeight=22 IconWidth=20
IndexOptions IgnoreClient NameWidth=40 DescriptionWidth=* XHTML FoldersFirst
 
# don't show these files and folders
IndexIgnore .htaccess .ht* *_notes *.log feed inc HEADER.html FOOTER.html feed*.gif
 
# the SSI files used for the header and footer
HeaderName /ssi/HEADER.html
ReadmeName /ssi/FOOTER.html
 
# used to determine the time and for SSI output
SetEnv TZ America/Indianapolis
SetEnv SERVER_ADMIN webmaster@askapache.com

In your /ssi/.htaccess

# makes files ending in .html be filtered through the INCLUDES filter before being sent to client
AddOutputFilter Includes html

My HEADER.html

<!--#set var="PAGETITLE" value="-- z.askapache.com" -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
  <title><!--#echo encoding="none" var="REQUEST_URI" --> <!--#echo encoding="none" var="PAGETITLE" --></title>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <link rel="stylesheet" href="http://z.askapache.com/c/error.css" />
  <link rev="made" href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->" />
</head>
<body>
<h1><a href="http://www.askapache.com/" title="AskApache Home" accesskey="1">
<img src="http://www.askapache.com/nlogo.jpg" height="75" alt="AskApache" />
</a> <!--#echo encoding="url" var="REQUEST_URI" --> <!--#echo encoding="none" var="PAGETITLE" --></h1>
<hr />

My FOOTER.html

<p>Find the information you are looking for on the <a href="http://www.askapache.com/">AskApache Home page</a>
or the <a href="http://www.askapache.com/search<!--#echo encoding="url" var="REQUEST_URI" -->">
AskApache search page</a>.</p>
<hr />
<address>
  <small>$Id:<!--#echo encoding="none" var="UNIQUE_ID" --> E:<!--#echo encoding="none" var="REDIRECT_STATUS" -->,v 1.30
<!--#config timefmt="%c" --><!--#echo var="DATE_LOCAL" --></small><br />
  <small><!--#echo var="SERVER_SOFTWARE" --></small><br /><br />
  z.askapache.com -- AskApache | <a href="http://www.askapache.com/about/">Webmaster</a>
| Copyright © 2009 AskApache<br />
</address>
<!--#if expr="$REMOTE_ADDR = 10.10.10.10" -->
<pre ><!--#printenv --></ pre>
<!--#endif -->
<script src="http://z.askapache.com/j/apache-0780.js" type="text/javascript"></script>
<script type="text/javascript">var pageTracker = _gat._getTracker("UA-7"+"321"+"53-38");
pageTracker._initData();pageTracker._trackPageview();</script>
</body>
</html>

Enhanced Error Pages

If you are using WordPress, I’m sure you are using my AskApache Google 404 Plugin, but whatever your error pages may be, using SSI you can make them better. My 404 Error Page is handled by WordPress and PHP, but all of my other error pages (codes 4xx to 5xx) are handled using SSI. You can check out any of them: /show-error-400, /show-error-403, /show-error-503, etc..

Notice the email note which has the subject prefilled? Thats one of the main uses for SSI, you can add forms to your errordocuments and get notified of problems which mean you can fix them.

Add this to your /.htaccess for each ErrorDocument you make.

ErrorDocument 503 /errordocs/503.html

My /errordocs/.htaccess

# turn on symlinks for rewrites and turn off SSI's ability to exec
Options None
Options SymLinksIfOwnerMatch IncludesNOEXEC
 
# makes files ending in .html be filtered through the INCLUDES filter before being sent to client
AddOutputFilter Includes html
 
# this internal apache variable prevents your errordocs from allowing keep-alive connections
SetEnv nokeepalive
 
# used to determine the time and for SSI output
SetEnv TZ America/Indianapolis
SetEnv SERVER_ADMIN webmaster@askapache.com

My /errordocs/503.html

<!--#set var="TITLE" value="Service Temporarily Unavailable" -->
<!--#include virtual="/errordocs/TOP.html" -->
<p>The server is temporarily unable to service your
request due to <strong>maintenance downtime</strong>
or super-crazy-extreme capacity problems. Please try
again later... Or <a href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->">send me an email</a> and let me know about it..</p>
<!--#include virtual="/errordocs/BOTTOM.html" -->

My /errordocs/TOP.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
  <title><!--#echo encoding="none" var="REQUEST_URI" --> <!--#echo encoding="none" var="TITLE" --></title>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <link rel="stylesheet" href="http://z.askapache.com/c/error.css" />
  <link rev="made" href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->" />
</head>
<body>
<h1><a href="http://www.askapache.com/" title="AskApache Home" accesskey="1"><img src="http://www.askapache.com/nlogo.jpg" height="75" alt="AskApache" />
</a> <!--#echo encoding="url" var="REQUEST_URI" --> <!--#echo encoding="none" var="TITLE" --></h1>
<hr />

My /errordocs/BOTTOM.html

<p>If this should not be an error please <a href="mailto:webmaster@askapache.com?subject=ID#<!--#echo encoding="none" var="UNIQUE_ID" -->">email</a> me right away.</p>
<!--#if expr="$HTTP_REFERER" -->
<p>You came from <!--#echo var="HTTP_REFERER"--></p>
<!--#endif -->
<p>If you still have a question, please try to find the information you are looking
for on the <a href="http://www.askapache.com/">AskApache Home page</a> or
the <a href="http://www.askapache.com/search/">AskApache search page</a>.</p>
<hr />
<address>
  <a href="http://www.askapache.com/about/">Webmaster</a>
  $Id:<!--#echo encoding="none" var="UNIQUE_ID" --> Error-<!--#echo encoding="none" var="REDIRECT_STATUS" -->,v 1.30 <!--#config timefmt="%c" --><!--#echo var="DATE_LOCAL" --><br />
  <!--#echo var="SERVER_SOFTWARE" -->
</address>
<!--#if expr="$REMOTE_ADDR = 10.10.10.10" -->
<pre ><!--#printenv --></ pre>
<!--#endif -->
</body>
</html>

Dealing with Spam and Website Attacks

Say you are using some nice .htaccess rewrite rules to block offending bots, web scrapers, and other nefarious net characters. Instead of just sending a 403 Denied, you could send them to be handled by an SSI document that could do any number of things.. From adding the offending bot’s IP address to the .htaccess Deny List (blacklisting), emailing you with an alert, emailing the IP Block Owner, executing a denial-of-service response ala guardian (script will return an artificially high Content-Length, and will then spoon-feed content bytes back to the client at a rate of one byte per second, for single-threaded or fixed-threadpool clients, this will hang all of their requests and render the attack inoperable), or just output a frightening looking message which usually does the trick if the bot is humanoid.

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\=?(http|ftp|ssl|https):/.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\?.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\.(asp|ini|dll).*\ HTTP/ [NC]
RewriteRule .* - [F]
 
ErrorDocument 403 /errordocs/f-off.html

Example /errordocs/f-off.html View it Live (Not Responsible for mental or physical harm caused by fright)

<html><head><title>WARNING <!--#echo encoding="none" var="REMOTE_ADDR" -->
- SECURITY TEAM ALERTED</title></head><body>
<h2>ATTENTION <!--#echo encoding="none" var="REMOTE_ADDR" --></h2>
<p>You have been flagged by our system as a potential threat.</p>
<p>This request has been logged and the Security Team has been notified.<br>
Repeated attempts from IP address <strong><!--#echo encoding="none" var="REMOTE_ADDR" --></strong> will result <br>
in the IP automatically being <strong>blacklisted</strong> on this server<br>
and sent to blacklists around the world.  Additionally, the Security Team<br>
WILL alert the IP Address Block Owner and/or contact the authorities.<br><br></p>
 
<h3>LOGGED INFORMATION FROM THIS REQUEST</h3>
< pre>
REMOTE_ADDR   :  <!--#echo encoding="none" var="REMOTE_ADDR" -->
REMOTE_PORT   :  <!--#echo encoding="none" var="REMOTE_PORT" -->
REQUEST_METHOD:  <!--#echo encoding="none" var="REQUEST_METHOD" --><!--#if expr="$HTTP_REFERER"
-->
REFERER       :  <!--#echo encoding="url" var="HTTP_REFERER" --><!--#endif
-->
REQUEST_URI   :  <!--#echo encoding="url" var="REQUEST_URI" --><!--#if expr="$HTTP_USER_AGENT"
-->
USER_AGENT    :</ pre>
< pre style="white-space:normal;margin-left:150px;max-width:80%;">  <!--#echo encoding="none" var="HTTP_USER_AGENT" -->
</ pre>
< pre><!--#endif
--><!--#if expr="$HTTP_COOKIE"
-->
COOKIES       :</ pre>
< pre style="white-space:normal;margin-left:150px;max-width:80%;">  <!--#echo encoding="none" var="HTTP_COOKIE" -->
</ pre>
< pre><!--#endif
--><!--#if expr="$QUERY_STRING"
-->
QUERY_STRING  :  <!--#echo encoding="url" var="QUERY_STRING" --><!--#endif
-->
</ pre></body></html>

Server Side Includes Detailed Info

Ok now that we have the real-world usage out of the way, lets dig in a bit to the actual module mod_include, which if you want you can view the source code here.

Enabling Server-Side Includes

Server Side Includes are implemented by the INCLUDES filter. For backwards compatibility, the server-parsed handler also activates the INCLUDES filter. As well, Apache will activate the INCLUDES filter for any document with mime type text/x-server-parsed-html or text/x-server-parsed-html3 (and the resulting output will have the mime type text/html). If documents containing server-side include directives are given the extension .shtml, the following directives will make Apache parse them and assign the resulting document the mime type of text/html:

AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

The following directive must be given for the directories containing the shtml files (typically in a section, but this directive is also valid in .htaccess files if AllowOverride Options is set):

Options +Includes

Server-Side Include Directives

These are the Directives allowed in .htaccess files that are handled by mod_include. Note that other modules may add additional directives, for instance the exec SSI Directive is supplied by mod_cgi. This is how mod_cgi registers with mod_include to provide processing of the exec directive. This is the code required to handle the “exec” SSI directive.

cgi_pfn_reg_with_ssi = APR_RETRIEVE_OPTIONAL_FN(ap_register_include_handler);
cgi_pfn_reg_with_ssi("exec", handle_exec);
static const char * const aszPre[] = { "mod_include.c", NULL };

config

Controls various aspects of the parsing.

<!--#config [timefmt="..."] [sizefmt="..."] [errmsg="..."] -->
<!--#config [echomsg="..."] -->
<!--#config errmsg="[It appears that you don't know how to use SSI]" -->

echomsg

(since 2.1) The value is a message that is sent back to the client if the <a href="#element.echo">echo</a> element attempts to echo an undefined variable. This overrides any <a href="#ssiundefinedecho">SSIUndefinedEcho</a> directives.

errmsg

The value is a message that is sent back to the client if an error occurs while parsing the document. This overrides any <a href="#ssierrormsg">SSIErrorMsg</a> directives.

sizefmt

The value sets the format to be used which displaying the size of a file. Valid values are bytes for a count in bytes, or abbrev for a count in Kb or Mb as appropriate, for example a size of 1024 bytes will be printed as “1K”.

timefmt

The value is a string to be used by the strftime(3) library routine when printing dates.

echo

Prints one of the include variables. If the variable is unset, the result is determined by the SSIUndefinedEcho directive.

<!--#echo [encoding="none|url|entity"] var="..." [encoding="none|url|entity"] var="..." ... -->

var

The value is the name of the variable to print.

encoding

Specifies how Apache should encode special characters contained in the variable before outputting them. If set to none, no encoding will be done. If set to url, then URL encoding (also known as %-encoding; this is appropriate for use within URLs in links, etc.) will be performed. The default is set to entity, resulting in entity encoding.

exec

Execute external programs

<!--#exec cgi="/cgi-bin/s.cgi" -->
<!--#exec cmd="ls" -->
<!--#include virtual="/cgi-bin/s.cgi?argument=value" -->
<!--#exec cmd="perl /cgi-bin/s.pl args" -->

include

Include a file

<!--#include virtual|file="..." [virtual|file="..."] ... -->

printenv

Print all available variables

<!--#printenv -->

set

Set a value of a variable.

<!--#set var="..." value="..." ... -->
<!--#set var="modified" value="$LAST_MODIFIED" -->
<!--#set var="name" value="AskApache" -->
<!--#set var="date" value="${DATE_LOCAL}_${DATE_GMT}" -->

flastmod

Prints the last modification date of the specified file, subject to the timefmt format specification.

<!--#flastmod virtual|file="..." [virtual|file="..."] ... -->

fsize

Prints the size of the specified file, subject to the sizefmt format specification.

<!--#fsize virtual|file="..." [virtual|file="..."] ... -->

if

The if element works like an if statement in a programming language. The test condition is evaluated and if the result is true, then the text until the next elif, else or endif element is included in the output stream.

<!--#if expr="..." -->
<!--#if expr="${REMOTE_USER} && ${HTTP_USER_AGENT}" -->

<!--#if expr="test_condition" -->
<!--#elif expr="test_condition" -->
<!--#else -->
<!--#endif -->

elif

Used to put text into the output stream if the original test_condition was false.

<!--#elif expr="..." -->

else

Used to put text into the output stream if the original test_condition was false.

<!--#else -->

endif

Ends the if element and is required.

<!--#endif -->

.htaccess directives

1. XBitHack
2. SSIErrorMsg
3. SSITimeFormat
4. SSIStartTag
5. SSIEndTag
6. SSIUndefinedEcho
7. SSIAccessEnable

mod_include Default SSI Values

START_SEQUENCE

<!--# – The starting tag for mod_include to recognize and parse as SSI.

END_SEQUENCE

--> - The ending tag for mod_include to recognize and parse as SSI.

ERROR_MSG

[an error occurred while processing this directive] - On Errors parsing SSI.

TIME_FORMAT

%A, %d-%b-%Y %H:%M:%S %Z - Default Time format for DATE

UNDEFINED_ECHO

(none) - When echoing an undefined variable.

SSI Variables

DATE_GMT=Sun Mar  8 22:58:56 2009
DATE_LOCAL=Sun Mar  8 15:58:56 2009
DOCUMENT_NAME=FOOTER.html
DOCUMENT_ROOT=/root-srv/protected/askapache.com/sec
DOCUMENT_URI=/includes/FOOTER.html
GATEWAY_INTERFACE=CGI/1.1
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_ACCEPT_ENCODING=gzip,deflate
HTTP_ACCEPT_LANGUAGE=en-us,en;q=0.5
HTTP_CACHE_CONTROL=max-age=0
HTTP_CONNECTION=keep-alive
HTTP_COOKIE=__qca=12298910-686528-46510;  __utmb=50625.1.0.11311
HTTP_HOST=www.askapache.com
HTTP_KEEP_ALIVE=300
HTTP_REFERER=http://www.askapache.com/htaccess/htaccess.html
HTTP_USER_AGENT=Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)
LAST_MODIFIED=Sun Mar  8 14:53:50 2009
PATH=/bin:/usr/bin:/sbin:/usr/sbin
QUERY_STRING=
REMOTE_ADDR=24.123.215.60
REMOTE_PORT=4785
REQUEST_METHOD=GET
REQUEST_URI=/htaccess/
SCRIPT_FILENAME=/root-srv/protected/askapache.com/sec/includes/FOOTER.html
SCRIPT_NAME=/includes/FOOTER.html
SCRIPT_URI=http://www.askapache.com/htaccess/
SCRIPT_URL=/htaccess/
SERVER_ADDR=64.111.114.111
SERVER_ADMIN=webmaster@askapache.com
SERVER_NAME=www.askapache.com
SERVER_PORT=80
SERVER_PROTOCOL=INCLUDED
SERVER_SIGNATURE=
SERVER_SOFTWARE=Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.63 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.4.2
UNIQUE_ID=dnbtH0Bvcm8A2ZHqcAAAAM
USER_NAME=

More SSI Information

1. mod_include
2. Apache Filters
3. Introduction to Server Side Includes
4. Apache Handlers

Advanced Htaccess – SSI, ErrorDocuments, DirectoryIndexing SEO originally appeared on AskApache.