Dealing with Mobile Visitors using Bad Browsers originally appeared on AskApache.com

Ok, sup. I really felt I had to get this out of the way, because I have a whole stack of drafts waiting to be published, but I realized that not many people will benefit from all the advanced optimizations and tricks I'm writing unless they get a basic understanding of some of the tools I'm using. I decided to write a series of articles explaining how I optimize servers for speed because lately I've been getting a lot more people wanting to hire me to do that. I take on projects when I can but there is clearly a need out here on the net for some self-help. The momentum is swinging more and more towards VPS type of web hosting, and I would say that 99% of those customers are getting supremely ripped off, which goes against the foundation of the web.

Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I'm not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn't it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won't be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU's handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore "processes") are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place... because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I've opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that's completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 - tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching - CPU processing
2. DISK IO Bottleneck - Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 - Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it's important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N - add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not "punchy" enough), the scheduler was decoupled from 'time slice' and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization "split" between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be "multiplicative" (or exponential) - that way it does not matter which nice level you start out from, the 'relative result' will always be the same.

The third complaint (of negative nice levels not being "punchy" enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID's 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following "real-time" policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following "normal" (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn't set a specific io priority. This is the default scheduling class for any process that hasn't asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it's directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn't really apply here.

ionice

ionice - get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c - The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n - The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p - Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs 'bash' as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]
 
-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities - schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces -- thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I've found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn't be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit - get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell's resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft' resource limit

-H

use the `hard' resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process's data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft', `hard', and `unlimited' stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process's data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process's nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 - rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process's resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler's nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on AskApache.com

If you have files on your site that you don't want indexed by malicious search engines, grabbed and leeched by malicious spammers, or stolen and made available elsewhere, you can use mod_rewrite to drastically reduce or totally reduce that activity.

The Worst Kind of People

Spammers, and Leechers. They operate like this: Let's say you have some mp3 files on a server, and SOMEWHERE on the web there is a link to that mp3 file's location. This includes in javascript files, css files, robots.txt files, the spammers and leechers robots check all those files looking for the type of link they are looking for. Then they try to request that file usually utilizing a number of different types of requests to get access to the file. Then they use it for personal gain, at your peril.

Some robots perform valuable services for the world wide web community, and other leeching programming is pretty cool, so not all these activities are perpetrated by nefarious spammers.

Blocking by IP COOKIE

Ok so if a link exists to your file, it is going to be requested by a robot eventually, so the way to defeat them is by doing something on your site that modifies the way a user would request it. Robots for the most part are not javascript-capable, so the most-common advanced method is to set a cookie using javascript, and then we can check for that cookie in the request for the file using mod_rewrite.

So if your site sets a cookie named fspammers, and furthermore gives that cookie a value of 445, then this is what the request sent by an HTTP Client like Firefox looks like.

GET /hotlink/lovefreedom.mp3 HTTP/1.1
Host: static.askapache.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.askapache.com/wordpress/seo-in-wordpress.html
Cookie: fspammers=455

Mod_Rewrite HTTP Headers

The mod_rewrite module has access to ALL the HTTP Headers sent in a request, so for each of the HTTP Headers in the request example above, we can use mod_rewrite to validate.

Mod_Rewrite .htaccess Example

Finally, now that everyone is on the same page about what is really going on, here is the .htaccess code that blocks any requests for anything in the /hotlink/ folder.

Here are the triggers this code blocks access based on.

1. Cookie: Checks if fspammers cookie is present, and that it has the value of 445.
2. HTTP Protocol: Checks if HTTP 1.1 is being used (many robots use 1.0)
3. Host: Checks that the HOST Requested was static.askapache.com
4. Referer: Checks for Referring site is static.askapache.com or www.askapache.com

RewriteEngine On
RewriteBase /
 
RewriteCond %{HTTP_COOKIE} !^.*fspammers=445.*$ [NC,OR]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /(.*)\ HTTP/1\.1 [NC,OR]
RewriteCond %{HTTP_HOST} !^z\.askapache\.com$ [NC,OR]
RewriteCond %{HTTP_REFERER} !^http://(www|z)\.askapache\.com.*$ [NC]
RewriteRule ^hotlink/.*$ - [F]</p>

Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking originally appeared on AskApache.com

.htaccess is a very ancient configuration file that controls the Web Server running your website, and is one of the most powerful configuration files you will ever come across. Htaccess has the ability to control access of the WWW's HyperText Transfer Protocol (HTTP) using Password Protection, 301 Redirects, rewrites, and much much more. This is because this configuration file was coded in the earliest days of the web (HTTP), for one of the first Web Servers ever! Eventually these Web Servers (configured with htaccess) became known as the World Wide Web, and eventually grew into the Internet we use today.

This is not an introduction to .htaccess… This is the evolution of the best of the best .htaccess on the planet. Its changed over the years with more and better .htaccess tricks using real .htaccess example code. I add my favorite htaccess-related links and sites, results from my htaccess research, and frequently update this article.

You've come to the right place if you are looking to acquire mad skills for using .htaccess files.

Originally (2003) this guide was known in certain hacker circles and hidden corners of the net as an ultimate .htaccess due to the powerful htaccess tricks and tips to bypass security on a webhost, and also because many of the tricks and examples were pretty impressive back then in that group.

AskApache Htaccess Journey

Skip this - still under edit

I discovered these tips and tricks mostly while working as a network security penetration specialist hired to find security holes in web hosting environments. Shared hosting is the most common and cheapest form of web-hosting where multiple customers are placed on a single machine and "share" the resources (CPU/RAM/SPACE). The machines are configured to basically ONLY do HTTP and FTP. No shells or any interactive logins, no ssh, just FTP access. That is when I started examining htaccess files in great detail and learned about the incredible untapped power of htaccess. For 99% of the worlds best Apache admins, they don't use .htaccess much, if AT ALL. It's much easier, safer, and faster to configure Apache using the httpd.conf file instead. However, this file is almost never readable on shared-hosts, and I've never seen it writable. So the only avenue left for those on shared-hosting was and is the .htaccess file, and holy freaking fiber-optics.. it's almost as powerful as httpd.conf itself!

Most all .htaccess code works in the httpd.conf file, but not all httpd.conf code works in .htaccess files, around 50%. So all the best Apache admins and programmers never used .htaccess files. There was no incentive for those with access to httpd.conf to use htaccess, and the gap grew. It's common to see "computer gurus" on forums and mailing lists rail against all uses and users of .htaccess files, smugly announcing the well known problems with .htaccess files compared with httpd.conf - I wonder if these "gurus" know the history of the htaccess file, like it's use in the earliest versions of the HTTP Server- NCSA's HTTPd, which BTW, became known as Apache HTTP. So you could easily say that htaccess files predates Apache itself.

Once I discovered what .htaccess files could do towards helping me enumerate and exploit security vulnerabilities even on big shared-hosts I focused all my research into .htaccess files, meaning I was reading the venerable Apache HTTP Source code 24/7! I compiled every released version of the Apache Web Server, ever, even NCSA's, and focused on enumerating the most powerful htaccess directives. Good times! Because my focus was on protocol/file/network vulnerabilites instead of web dev I built up a nice toolbox of htaccess tricks to do unusual things. When I switched over to webdev in 2005 I started using htaccess for websites, not research. I documented most of my favorites and rewrote the htaccess guide for webdevelopers. After some great encouragement on various forums and nets I decided to start a blog to share my work with everyone, AskApache.com was registered, I published my guide, and it was quickly plagiarized and scraped all over the net. Information is freedom, and freedom is information, so this blog has the least restrictive copyright for you. Feel free to modify, copy, republish, sell, or use anything on this site ;)

Htaccess - Evolved

The Hyper Text Transfer Protocol (HTTP) was initiated at the CERN in Geneve (Switzerland), where it emerged (together with the HTML presentation language) from the need to exchange scientific information on a computer network in a simple manner. The first public HTTP implementation only allowed for plain text information, and almost instantaneously became a replacement of the GOPHER service. One of the first text-based browsers was LYNX which still exists today; a graphical HTTP client appeared very quickly with the name NCSA Mosaic. Mosaic was a popular browser back in 1994. Soon the need for a more rich multimedia experience was born, and the markup language provided support for a growing multitude of media types.

Htaccess file know-how will do several things for you:

• Make your website noticeably faster.
• Allow you to debug your server with ease.
• Make your life easier and more rewarding.
• Allow you to work faster and more productively.

What Is .htaccess

Specifically, .htaccess is the default file name of a special configuration file that provides a number of directives (commands) for controlling and configuring the Apache Web Server, and also to control and configure modules that can be built into the Apache installation, or included at run-time like mod_rewrite (for htaccess rewrite), mod_alias (for htaccess redirects), and mod_ssl (for controlling SSL connections).

Htaccess allows for decentralized management of Web Server configurations which makes life very easy for web hosting companies and especially their savvy consumers. They set up and run "server farms" where many hundreds and thousands of web hosting customers are all put on the same Apache Server. This type of hosting is called "virtual hosting" and without .htaccess files would mean that every customer must use the same exact settings as everyone else on their segment. So that is why any half-decent web host allows/enables (DreamHost, Powweb, MediaTemple, GoDaddy) .htaccess files, though few people are aware of it. Let's just say that if I was a customer on your server-farm, and .htaccess files were enabled, my websites would be a LOT faster than yours, as these configuration files allow you to fully take advantage of and utilize the resources allotted to you by your host. If even 1/10 of the sites on a server-farm took advantage of what they are paying for, the providers would go out of business.

One of the design goals for this server was to maintain external compatibility with the NCSA 1.3 server --- that is, to read the same configuration files, to process all the directives therein correctly, and in general to be a drop-in replacement for NCSA. On the other hand, another design goal was to move as much of the server's functionality into modules which have as little as possible to do with the monolithic server core. The only way to reconcile these goals is to move the handling of most commands from the central server into the modules.

However, just giving the modules command tables is not enough to divorce them completely from the server core. The server has to remember the commands in order to act on them later. That involves maintaining data which is private to the modules, and which can be either per-server, or per-directory. Most things are per-directory, including in particular access control and authorization information, but also information on how to determine file types from suffixes, which can be modified by AddType and DefaultType directives, and so forth. In general, the governing philosophy is that anything which can be made configurable by directory should be; per-server information is generally used in the standard set of modules for information like Aliases and Redirects which come into play before the request is tied to a particular place in the underlying file system.

Another requirement for emulating the NCSA server is being able to handle the per-directory configuration files, generally called .htaccess files, though even in the NCSA server they can contain directives which have nothing at all to do with access control. Accordingly, after URI -> filename translation, but before performing any other phase, the server walks down the directory hierarchy of the underlying filesystem, following the translated pathname, to read any .htaccess files which might be present. The information which is read in then has to be merged with the applicable information from the server's own config files (either from the <directory> sections in access.conf, or from defaults in srm.conf, which actually behaves for most purposes almost exactly like <directory />).

Finally, after having served a request which involved reading .htaccess files, we need to discard the storage allocated for handling them. That is solved the same way it is solved wherever else similar problems come up, by tying those structures to the per-transaction resource pool.

Creating Htaccess Files

Htaccess files use the default filename ".htaccess" but any unix-style file name can be specified from the main server config using the AccessFileName directive. The file isn't .htaccess.txt, its literally just named .htaccess.

In a Windows Environment like the one I use for work, you can change how Windows opens and views .htaccess files by modifying the Folder Options in explorer. As you can see, on my computer files ending in .htaccess are recognized as having the HTACCESS extension and are handled/opened by Adobe Dreamweaver CS4.

Htaccess Scope

Unlike the main server configuration files like httpd.conf, Htaccess files are read on every request therefore changes in these files take immediate effect. Apache searches all directories and subdirectories that are htaccess-enabled for an .htaccess file which results in performance loss due to file accesses. I've never noticed a performance loss but OTOH, I know how to use them. If you do have access to your main server configuration file, you should of course use that instead, and lucky for you ALL the .htaccess tricks and examples can be used there as well (just not vice versa).

Htaccess File Syntax

Htaccess files follow the same syntax as the main Apache configuration files, for powerusers here's an apache.vim for VI. The one main difference is the context of the directive, which means whether or not that directive is ALLOWED to be used inside of an .htaccess file. Htaccess files are incredibly powerful, and can also be very dangerous as some directives allowed in the main configuration files would allow users/customers to completely bypass security/bandwidth-limits/resource-limits/file-permissions, etc.. About 1/4 of all Apache directives cannot be used inside an .htaccess file (also known as a per-directory context config). The Apache Developers are well-regarded throughout the world as being among some of the best programmers, ever. To enable a disallowed directive inside a .htaccess file would require modifying the source code and re-compiling the server (which they allow and encourage if you are the owner/admin). Here's a taste of that famous Apache source code that builds the directives allowed in .htaccess file context, the key that tells whether its enabled in .htaccess context is the DIR_CMD_PERMS and then the OR_FILEINFO, which means a directive is enabled dependent on the AllowOverride directive that is only allowed in the main config. First Apache 1.3.0, then Apache 2.2.10

mod_autoindex

AddIcon, add_icon, BY_PATH, DIR_CMD_PERMS, an icon URL followed by one or more filenames
AddIconByType, add_icon, BY_TYPE, DIR_CMD_PERMS, an icon URL followed by one or more MIME types
AddIconByEncoding, add_icon, BY_ENCODING, DIR_CMD_PERMS, an icon URL followed by one or more content encodings
AddAlt, add_alt, BY_PATH, DIR_CMD_PERMS, alternate descriptive text followed by one or more filenames
AddAltByType, add_alt, BY_TYPE, DIR_CMD_PERMS, alternate descriptive text followed by one or more MIME types
AddAltByEncoding, add_alt, BY_ENCODING, DIR_CMD_PERMS, alternate descriptive text followed by one or more content encodings
IndexOptions, add_opts, DIR_CMD_PERMS, RAW_ARGS, one or more index options
IndexIgnore, add_ignore, DIR_CMD_PERMS, ITERATE, one or more file extensions
AddDescription, add_desc, BY_PATH, DIR_CMD_PERMS, Descriptive text followed by one or more filenames
HeaderName, add_header, DIR_CMD_PERMS, TAKE1, a filename
ReadmeName, add_readme, DIR_CMD_PERMS, TAKE1, a filename
FancyIndexing, fancy_indexing, DIR_CMD_PERMS, FLAG, Limited to 'on' or 'off' (superseded by IndexOptions FancyIndexing)
DefaultIcon, ap_set_string_slot, (void *) XtOffsetOf(autoindex_config_rec, default_icon), DIR_CMD_PERMS, TAKE1, an icon URL

mod_rewrite

// mod_rewrite
RewriteEngine, cmd_rewriteengine, OR_FILEINFO, On or Off to enable or disable (default)
RewriteOptions, cmd_rewriteoptions, OR_FILEINFO, List of option strings to set
RewriteBase, cmd_rewritebase, OR_FILEINFO, the base URL of the per-directory context
RewriteCond, cmd_rewritecond, OR_FILEINFO, an input string and a to be applied regexp-pattern
RewriteRule, cmd_rewriterule, OR_FILEINFO, an URL-applied regexp-pattern and a substitution URL
RewriteMap, cmd_rewritemap, RSRC_CONF, a mapname and a filename
RewriteLock, cmd_rewritelock, RSRC_CONF, the filename of a lockfile used for inter-process synchronization
RewriteLog, cmd_rewritelog, RSRC_CONF, the filename of the rewriting logfile
RewriteLogLevel, cmd_rewriteloglevel, RSRC_CONF, the level of the rewriting logfile verbosity (0=none, 1=std, .., 9=max)
RewriteLog, fake_rewritelog, RSRC_CONF, [DISABLED] the filename of the rewriting logfile
RewriteLogLevel, fake_rewritelog, RSRC_CONF, [DISABLED] the level of the rewriting logfile verbosity

Htaccess Directives

Don't ask why, but I personally downloaded each major/beta release of the Apache HTTPD source code from version 1.3.0 to version 2.2.10 (all 63 Apache versions!), then I configured and compiled each version for a custom HTTPD installation built from source. This allowed me to find every directive allowed in .htaccess files for each particular version, which has never been done before, or since. YES! I think that is so cool..

An .htaccess directive is basically a command that is specific to a module or builtin to the core that performs a specific task or sets a specific setting for how Apache serves your WebSite. Directives placed in Htaccess files apply to the directory they are in, and all sub-directories. Here's the 3 top links (official Apache Docs) you will repeatedly use, bookmark/print/save them.

1. Terms Used to Describe Directives
2. Official List of Apache Directives
3. Directive Quick-Reference -- with Context

Litespeed Htaccess support

Unlike other lightweight web servers, Apache compatible per-directory configuration overridden is fully supported by LiteSpeed Web Server. With .htacess you can change configurations for any directory under document root on-the-fly, which in most cases is a mandatory feature in shared hosting environment. It is worth noting that enabling .htaccess support in LiteSpeed Web Server will not degrade server's performance, comparing to Apache's 40% drop in performance.

Main Server Config Examples

Now lets take a look at some htaccess examples to get a feel for the syntax and some general ideas at the capabilities. Some of the best examples for .htaccess files are included with Apache for main server config files, so lets take a quick look at a couple of them on our way down to the actual .htaccess examples further down the page (this site has thousands, take your time). As you can see, the basic syntax is a line starting with # is a comment, everything else are directives followed by the directive argument.

httpd-multilang-errordoc.conf: The configuration below implements multi-language error documents through content-negotiation

Options IncludesNoExec
AddOutputFilter Includes html
AddHandler type-map var
LanguagePriority en cs de es fr it ja ko nl pl pt-br ro sv tr
ForceLanguagePriority Prefer Fallback
ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var

httpd-manual.conf: Provide local access to the server documentation on your server

SetEnvIf Request_URI ^/manual/(de|en|es|fr|ja|ko|pt-br|ru|tr)/ prefer-language=$1
RedirectMatch 301 ^/manual(?:/(de|en|es|fr|ja|ko|pt-br|ru|tr)){2, }(/.*)?$ /manual/$1$2
LanguagePriority en de es fr ja ko pt-br ru tr
ForceLanguagePriority Prefer Fallback

httpd-languages.conf: Settings for hosting different languages.

DefaultLanguage en
AddLanguage ca .ca
# Just list the languages in decreasing order of preference.
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv tr zh-CN zh-TW
# Commonly used filename extensions to character sets.
AddCharset us-ascii.ascii .us-ascii
AddCharset ISO-8859-1  .iso8859-1  .latin1

httpd-autoindex.conf: Directives controlling the display of server-generated directory listings.

# IndexOptions: Controls the appearance of server-generated directory listings.
IndexOptions FancyIndexing HTMLTable VersionSort
# AddIcon* directives tell the server which icon to show for different files or filename extensions.
AddIconByEncoding (CMP, /icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT, /icons/text.gif) text/*
AddIcon /icons/folder.gif ^^DIRECTORY^^
# DefaultIcon is which icon to show for files which do not have an icon explicitly set.
DefaultIcon /icons/unknown.gif
# AddDescription allows you to place a short description after a file in server-generated indexes.
AddDescription "GZIP compressed document" .gz
# ReadmeName is the name of the README file the server will look for by default, and append to directory listings.
ReadmeName README.html
# HeaderName is the name of a file which should be prepended to directory indexes.
HeaderName HEADER.html

Here are the rest of them if you wanna take a look. (httpd-mpm.conf, httpd-default.conf, httpd-ssl.conf, httpd-info.conf, httpd-vhosts.conf, httpd-dav.conf)

Example .htaccess Files

Here are some samples and examples taken from different .htaccess files I've used over the years. Specific solutions are farther down on this page and throughout the site.

# Set the Time Zone of your Server
SetEnv TZ America/Indianapolis
# ServerAdmin:  This address appears on some server-generated pages, such as error documents.
SetEnv SERVER_ADMIN webmaster@askapache.com
# Possible values for the Options directive are "None", "All", or any combination of:
#  Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
Options -ExecCGI -MultiViews -Includes -Indexes FollowSymLinks
# DirectoryIndex: sets the file that Apache will serve if a directory is requested.
DirectoryIndex index.html index.php /index.php
#
# Action lets you define media types that will execute a script whenever
# a matching file is called. This eliminates the need for repeated URL
# pathnames for oft-used CGI file processors.
# Format: Action media/type /cgi-script/location
# Format: Action handler-name /cgi-script/location
#
Action php5-cgi /bin/php.cgi
#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
AddHandler php-cgi .php .inc
# Commonly used filename extensions to character sets.
AddDefaultCharset UTF-8
# AddType allows you to add to or override the MIME configuration
AddType 'application/rdf+xml; charset=UTF-8' .rdf
AddType 'application/xhtml+xml; charset=UTF-8' .xhtml
AddType 'application/xhtml+xml; charset=UTF-8' .xhtml.gz
AddType 'text/html; charset=UTF-8' .html
AddType 'text/html; charset=UTF-8' .html.gz
AddType application/octet-stream .rar .chm .bz2 .tgz .msi .pdf .exe
AddType application/vnd.ms-excel .csv
AddType application/x-httpd-php-source .phps
AddType application/x-pilot .prc .pdb
AddType application/x-shockwave-flash .swf
AddType application/xrds+xml .xrdf
AddType text/plain .ini .sh .bsh .bash .awk .nawk .gawk .csh .var .c .in .h .asc .md5 .sha .sha1
AddType video/x-flv .flv
# AddEncoding allows you to have certain browsers uncompress information on the fly. Note: Not all browsers support this.
AddEncoding x-compress .Z
AddEncoding x-gzip .gz .tgz
# DefaultType: the default MIME type the server will use for a document.
DefaultType text/html
#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off

## MAIN DEFAULTS
Options +ExecCGI -Indexes
DirectoryIndex index.html index.htm index.php
DefaultLanguage en-US
AddDefaultCharset UTF-8
ServerSignature Off
## ENVIRONMENT VARIABLES
SetEnv PHPRC /webroot/includes
SetEnv TZ America/Indianapolis
 
SetEnv SERVER_ADMIN webmaster@askapache.com
## MIME TYPES
AddType video/x-flv .flv
AddType application/x-shockwave-flash .swf
AddType image/x-icon .ico
## FORCE FILE TO DOWNLOAD INSTEAD OF APPEAR IN BROWSER
# http://www.htaccesselite.com/addtype-addhandler-action-vf6.html
AddType application/octet-stream .mov .mp3 .zip
## ERRORDOCUMENTS
# http://askapache.com/htaccess/apache-status-code-headers-errordocument.html
ErrorDocument 400 /e400/
ErrorDocument 401 /e401/
ErrorDocument 402 /e402/
ErrorDocument 403 /e403/
ErrorDocument 404 /e404/
#
# Handlers be builtin, included in a module, or added with Action directive
# default-handler: default, handles static content (core)
#   send-as-is: Send file with HTTP headers (mod_asis)
#   cgi-script: treat file as CGI script (mod_cgi)
#    imap-file: Parse as an imagemap rule file (mod_imap)
#   server-info: Get server config info (mod_info)
#  server-status: Get server status report (mod_status)
#    type-map: type map file for content negotiation (mod_negotiation)
#  fastcgi-script: treat file as fastcgi script (mod_fastcgi)
#
# http://www.askapache.com/php/custom-phpini-tips-and-tricks.html
## PARSE AS CGI
AddHandler cgi-script .cgi .pl .spl
## RUN PHP AS APACHE MODULE
AddHandler application/x-httpd-php .php .htm
## RUN PHP AS CGI
AddHandler php-cgi .php .htm
## CGI PHP WRAPPER FOR CUSTOM PHP.INI
AddHandler phpini-cgi .php .htm
Action phpini-cgi /cgi-bin/php5-custom-ini.cgi
## FAST-CGI SETUP WITH PHP-CGI WRAPPER FOR CUSTOM PHP.INI
AddHandler fastcgi-script .fcgi
AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php5-wrapper.fcgi
## CUSTOM PHP CGI BINARY SETUP
AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php.cgi
## PROCESS SPECIFIC FILETYPES WITH CGI-SCRIPT
Action image/gif /cgi-bin/img-create.cgi
## CREATE CUSTOM HANDLER FOR SPECIFIC FILE EXTENSIONS
AddHandler custom-processor .ssp
Action custom-processor /cgi-bin/myprocessor.cgi
### HEADER CACHING
# http://www.askapache.com/htaccess/speed-up-sites-with-htaccess-caching.html
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico)$">
Header set Cache-Control "max-age=2592000"
</FilesMatch>
<FilesMatch "\.(js|css|pdf|swf)$">
Header set Cache-Control "max-age=604800"
</FilesMatch>
<FilesMatch "\.(html|htm|txt)$">
Header set Cache-Control "max-age=600"
</FilesMatch>
<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>
## ALTERNATE EXPIRES CACHING
# htaccesselite.com/d/use-htaccess-to-speed-up-your-site-discussion-vt67.html
ExpiresActive On
ExpiresDefault A604800
ExpiresByType image/x-icon A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType text/css A2592000
ExpiresByType text/html A300
<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
ExpiresActive Off
</FilesMatch>
## META HTTP-EQUIV REPLACEMENTS
<FilesMatch "\.(html|htm|php)$">
Header set imagetoolbar "no"
</FilesMatch>

Here are some default MOD_REWRITE code examples.

## REWRITE DEFAULTS
RewriteEngine On
RewriteBase /
## REQUIRE SUBDOMAIN
RewriteCond %{HTTP_HOST} !^$
RewriteCond %{HTTP_HOST} !^subdomain\.askapache\.com$ [NC]
RewriteRule ^/(.*)$ http://subdomain.askapache.com/$1 [L,R=301]
## SEO REWRITES
RewriteRule ^(.*)/ve/(.*)$ $1/voluntary-employee/$2 [L,R=301]
RewriteRule ^(.*)/hsa/(.*)$ $1/health-saving-account/$2 [L,R=301]
## WORDPRESS
RewriteCond %{REQUEST_FILENAME} !-f  # Existing File
RewriteCond %{REQUEST_FILENAME} !-d  # Existing Directory
RewriteRule . /index.php [L]
## ALTERNATIVE ANTI-HOTLINKING
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(subdomain\.)?askapache.com/.*$ [NC]
RewriteRule ^.*\.(bmp|tif|gif|jpg|jpeg|jpe|png)$ - [F]
## REDIRECT HOTLINKERS
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(subdomain\.)?askapache.com/.*$ [NC]
RewriteRule ^.*\.(bmp|tif|gif|jpg|jpeg|jpe|png)$ http://google.com [R]
## DENY REQUEST BASED ON REQUEST METHOD
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK|OPTIONS|HEAD)$ [NC]
RewriteRule ^.*$ - [F]
## REDIRECT UPLOADS
RewriteCond %{REQUEST_METHOD} ^(PUT|POST)$ [NC]
RewriteRule ^(.*)$ /cgi-bin/form-upload-processor.cgi?p=$1 [L,QSA]
## REQUIRE SSL EVEN WHEN MOD_SSL IS NOT LOADED
RewriteCond %{HTTPS} !=on [NC]
RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L]
### ALTERNATATIVE TO USING ERRORDOCUMENT
# http://www.htaccesselite.com/d/htaccess-errordocument-examples-vt11.html
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^.*$ /error.php [L]
## SEO REDIRECTS
Redirect 301 /2006/oldfile.html http://subdomain.askapache.com/newfile.html
RedirectMatch 301 /o/(.*)$ http://subdomain.askapache.com/s/dl/$1

Examples of protecting your files and securing with password protection.

#
# Require (user|group|valid-user) (username|groupname)
#
## BASIC PASSWORD PROTECTION
AuthType basic
AuthName "prompt"
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user
## ALLOW FROM IP OR VALID PASSWORD
Require valid-user
Allow from 192.168.1.23
Satisfy Any
## PROTECT FILES
<FilesMatch "\.(htaccess|htpasswd|ini|phps|fla|psd|log|sh)$">
Order Allow,Deny
Deny from all
</FilesMatch>
## PREVENT HOTLINKING
SetEnvIfNoCase Referer "^http://subdomain.askapache.com/" good
SetEnvIfNoCase Referer "^$" good
<FilesMatch "\.(png|jpg|jpeg|gif|bmp|swf|flv)$">
Order Deny,Allow
Deny from all
Allow from env=good
ErrorDocument 403 http://www.google.com/intl/en_ALL/images/logo.gif
ErrorDocument 403 /images/you_bad_hotlinker.gif
</FilesMatch>
## LIMIT UPLOAD FILE SIZE TO PROTECT AGAINST DOS ATTACK
#bytes, 0-2147483647(2GB)
LimitRequestBody 10240000
## MOST SECURE WAY TO REQUIRE SSL
# http://www.askapache.com/htaccess/apache-ssl-in-htaccess-examples.html
SSLOptions +StrictRequire
SSLRequireSSL
SSLRequire %{HTTP_HOST} eq "askapache.com"
ErrorDocument 403 https://askapache.com
## COMBINED DEVELOPER HTACCESS CODE-USE THIS
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico|js|css|pdf|swf|html|htm|txt)$">
Header set Cache-Control "max-age=5"
</FilesMatch>
AuthType basic
AuthName "Ooops! Temporarily Under Construction..."
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user      # password prompt for everyone else
Order Deny,Allow
Deny from all
Allow from 192.168.64.5   # Your, the developers IP address
Allow from w3.org      # css/xhtml check jigsaw.w3.org/css-validator/
Allow from googlebot.com   # Allows google to crawl your pages
Satisfy Any        # no password required if host/ip is Allowed
## DONT HAVE TO EMPTY CACHE OR RELOAD TO SEE CHANGES
ExpiresDefault A5 #If using mod_expires
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico|js|css|pdf|swf|html|htm|txt)$">
Header set Cache-Control "max-age=5"
</FilesMatch>
## ALLOW ACCESS WITH PASSWORD OR NO PASSWORD FOR SPECIFIC IP/HOSTS
AuthType basic
AuthName "Ooops! Temporarily Under Construction..."
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user      # password prompt for everyone else
Order Deny,Allow
Deny from all
Allow from 192.168.64.5   # Your, the developers IP address
Allow from w3.org      # css/xhtml check jigsaw.w3.org/css-validator/
Allow from googlebot.com   # Allows google to crawl your pages
Satisfy Any        # no password required if host/ip is Allowed

Example .htaccess Code Snippets

Here are some specific examples, this is the most popular section of this page. Updated frequently.

Redirect Everyone Except IP address to alternate page

ErrorDocument 403 http://www.yahoo.com/
Order deny,allow
Deny from all
Allow from 208.113.134.190

When developing sites

This lets google crawl the page, lets me access without a password, and lets my client access the page WITH a password. It also allows for XHTML and CSS validation! (w3.org)

AuthName "Under Development"
AuthUserFile /home/sitename.com/.htpasswd
AuthType basic
Require valid-user
Order deny,allow
Deny from all
Allow from 208.113.134.190 w3.org htmlhelp.com googlebot.com
Satisfy Any

Fix double-login prompt

Redirect non-https requests to https server and ensure that .htpasswd authorization can only be entered across HTTPS

SSLOptions +StrictRequire
SSLRequireSSL
SSLRequire %{HTTP_HOST} eq "askapache.com"
ErrorDocument 403 https://askapache.com

Set Timezone of the Server (GMT)

SetEnv TZ America/Indianapolis

Administrator Email for ErrorDocument

SetEnv SERVER_ADMIN webmaster@google.com

ServerSignature for ErrorDocument

ServerSignature off | on | email

Charset and Language headers

Article: Setting Charset in htaccess, and article by Richard Ishida

AddDefaultCharset UTF-8
DefaultLanguage en-US

Disallow Script Execution

Options -ExecCGI
AddHandler cgi-script .php .pl .py .jsp .asp .htm .shtml .sh .cgi

Deny Request Methods

RewriteCond %{REQUEST_METHOD} !^(GET|HEAD|OPTIONS|POST|PUT)
RewriteRule .* - [F]

Force "File Save As" Prompt

AddType application/octet-stream .avi .mpg .mov .pdf .xls .mp4

Show CGI Source Code

RemoveHandler cgi-script .pl .py .cgi
AddType text/plain .pl .py .cgi

Serve all .pdf files on your site using .htaccess and mod_rewrite with the php script.

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^(.+)\.pdf$  /cgi-bin/pdf.php?file=$1 [L,NC,QSA]

Rewrite to www

RewriteCond %{REQUEST_URI} !^/(robots\.txt|favicon\.ico|sitemap\.xml)$
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Rewrite to www dynamically

RewriteCond %{REQUEST_URI} !^/robots\.txt$ [NC]
RewriteCond %{HTTP_HOST} !^www\.[a-z-]+\.[a-z]{2,6} [NC]
RewriteCond %{HTTP_HOST} ([a-z-]+\.[a-z]{2,6})$   [NC]
RewriteRule ^/(.*)$ http://%1/$1 [R=301,L]

301 Redirect Old File

Redirect 301 /old/file.html http://www.askapache.com/new/file.html

301 Redirect Entire Directory

RedirectMatch 301 /blog(.*) http://www.askapache.com/$1

Protecting your php.cgi

<FilesMatch "^php5?\.(ini|cgi)$">
Order Deny,Allow
Deny from All
Allow from env=REDIRECT_STATUS
</FilesMatch>

Set Cookie based on Request

This code sends the Set-Cookie header to create a cookie on the client with the value of a matching item in 2nd parantheses.

RewriteEngine On
RewriteBase /
RewriteRule ^(.*)(de|es|fr|it|ja|ru|en)/$ - [co=lang:$2:.askapache.com:7200:/]

Set Cookie with env variable

Header set Set-Cookie "language=%{lang}e; path=/;" env=lang

Custom ErrorDocuments

ErrorDocument 100 /100_CONTINUE
ErrorDocument 101 /101_SWITCHING_PROTOCOLS
ErrorDocument 102 /102_PROCESSING
ErrorDocument 200 /200_OK
ErrorDocument 201 /201_CREATED
ErrorDocument 202 /202_ACCEPTED
ErrorDocument 203 /203_NON_AUTHORITATIVE
ErrorDocument 204 /204_NO_CONTENT
ErrorDocument 205 /205_RESET_CONTENT
ErrorDocument 206 /206_PARTIAL_CONTENT
ErrorDocument 207 /207_MULTI_STATUS
ErrorDocument 300 /300_MULTIPLE_CHOICES
ErrorDocument 301 /301_MOVED_PERMANENTLY
ErrorDocument 302 /302_MOVED_TEMPORARILY
ErrorDocument 303 /303_SEE_OTHER
ErrorDocument 304 /304_NOT_MODIFIED
ErrorDocument 305 /305_USE_PROXY
ErrorDocument 307 /307_TEMPORARY_REDIRECT
ErrorDocument 400 /400_BAD_REQUEST
ErrorDocument 401 /401_UNAUTHORIZED
ErrorDocument 402 /402_PAYMENT_REQUIRED
ErrorDocument 403 /403_FORBIDDEN
ErrorDocument 404 /404_NOT_FOUND
 
ErrorDocument 405 /405_METHOD_NOT_ALLOWED
ErrorDocument 406 /406_NOT_ACCEPTABLE
ErrorDocument 407 /407_PROXY_AUTHENTICATION_REQUIRED
ErrorDocument 408 /408_REQUEST_TIME_OUT
ErrorDocument 409 /409_CONFLICT
ErrorDocument 410 /410_GONE
ErrorDocument 411 /411_LENGTH_REQUIRED
ErrorDocument 412 /412_PRECONDITION_FAILED
ErrorDocument 413 /413_REQUEST_ENTITY_TOO_LARGE
ErrorDocument 414 /414_REQUEST_URI_TOO_LARGE
ErrorDocument 415 /415_UNSUPPORTED_MEDIA_TYPE
ErrorDocument 416 /416_RANGE_NOT_SATISFIABLE
ErrorDocument 417 /417_EXPECTATION_FAILED
ErrorDocument 422 /422_UNPROCESSABLE_ENTITY
ErrorDocument 423 /423_LOCKED
ErrorDocument 424 /424_FAILED_DEPENDENCY
ErrorDocument 426 /426_UPGRADE_REQUIRED
ErrorDocument 500 /500_INTERNAL_SERVER_ERROR
ErrorDocument 501 /501_NOT_IMPLEMENTED
ErrorDocument 502 /502_BAD_GATEWAY
ErrorDocument 503 /503_SERVICE_UNAVAILABLE
ErrorDocument 504 /504_GATEWAY_TIME_OUT
ErrorDocument 505 /505_VERSION_NOT_SUPPORTED
ErrorDocument 506 /506_VARIANT_ALSO_VARIES
ErrorDocument 507 /507_INSUFFICIENT_STORAGE
ErrorDocument 510 /510_NOT_EXTENDED

Implementing a Caching Scheme with .htaccess

# year
<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|swf|mp3|mp4)$">
Header set Cache-Control "public"
Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
Header unset Last-Modified
</FilesMatch>
#2 hours
<FilesMatch "\.(html|htm|xml|txt|xsl)$">
Header set Cache-Control "max-age=7200, must-revalidate"
</FilesMatch>
<FilesMatch "\.(js|css)$">
SetOutputFilter DEFLATE
Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
</FilesMatch>

Password Protect single file

<Files login.php>
AuthName "Prompt"
AuthType Basic
AuthUserFile /home/askapache.com/.htpasswd
Require valid-user
</Files>

Password Protect multiple files

<FilesMatch "^(private|phpinfo)\.*$">
AuthName "Development"
AuthUserFile /.htpasswd
AuthType basic
Require valid-user
</FilesMatch>

Send Custom Headers

Header set P3P "policyref=\"http://www.askapache.com/w3c/p3p.xml\""
Header set X-Pingback "http://www.askapache.com/xmlrpc.php"
Header set Content-Language "en-US"
Header set Vary "Accept-Encoding"

Blocking based on User-Agent Header

SetEnvIfNoCase ^User-Agent$ .*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(libwww-perl|aesop_com_spiderman) HTTP_SAFE_BADBOT
Deny from env=HTTP_SAFE_BADBOT

Blocking with RewriteCond

RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
RewriteRule . - [F,L]

.htaccess for mod_php

SetEnv PHPRC /location/todir/containing/phpinifile

.htaccess for php as cgi

AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php5.cgi

Shell wrapper for custom php.ini

#!/bin/sh
export PHP_FCGI_CHILDREN=3
exec php5.cgi -c /abs/php5/php.ini

Add values from HTTP Headers

SetEnvIfNoCase ^If-Modified-Since$ "(.+)" HTTP_IF_MODIFIED_SINCE=$1
SetEnvIfNoCase ^If-None-Match$ "(.+)" HTTP_IF_NONE_MATCH=$1
SetEnvIfNoCase ^Cache-Control$ "(.+)" HTTP_CACHE_CONTROL=$1
SetEnvIfNoCase ^Connection$ "(.+)" HTTP_CONNECTION=$1
SetEnvIfNoCase ^Keep-Alive$ "(.+)" HTTP_KEEP_ALIVE=$1
SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1
SetEnvIfNoCase ^Cookie$ "(.+)" HTTP_MY_COOKIE=$1

Stop hotlinking

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?askapache.com/.*$ [NC]
RewriteRule \.(gif|jpg|swf|flv|png)$ http://www.askapache.com/feed.gif [R=302,L]

Turn logging off for IP

SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "nolog,noauditlog,pass"

Turn logging on for IP

SecFilterSelective REMOTE_ADDR "!^208\.113\.183\.103" "nolog,noauditlog,pass"
SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "log,auditlog,pass"

Advanced Mod_Rewrite Examples

Best .htaccess Articles

.htaccess for Webmasters

• Process certain requests for files using a cgi script
• Process Requests with certain Request Methods
• Make any file be a certain filetype
• Use IfModule directive for robust code

Mod_Rewrite URL Rewriting

Undocumented techniques and methods will allow you to utilize mod_rewrite at an "expert level" by showing you how to unlock its secrets.

• Check for a key in QUERY_STRING
• Block access to files during certain hours of the day
• Rewrite underscores to hyphens for SEO URL
• Redirecting WordPress Feeds to Feedburner

301 Redirects without mod_rewrite

• Redirect single url
• Redirect to new Domain

Secure PHP with .htaccess

If you have a php.cgi or php.ini file in your /cgi-bin/ directory or other pub directory, try requesting them from your web browser. If your php.ini shows up or worse you are able to execute your php cgi, you'll need to secure it ASAP. This shows several ways to secure these files, and other interpreters like perl, fastCGI, bash, csh, etc.

.htaccess Cookie Manipulation

Fresh .htaccess code for you! Check out the Cookie Manipulation and environment variable usage with mod_rewrite! I also included a couple Mod_Security .htaccess examples. Enjoy!

• Mod_Rewrite .htaccess Examples
• Cookie Manipulation and Tests with mod_rewrite
• Setting Environment Variables
• Using the Environment Variable
• Mod_Security .htaccess Examples

.htaccess Caching

• Speed Up Sites with htaccess Caching
• htaccess time cheat sheet

Password Protection and Authentication

• Require password for single file
• Example .htaccess file for password protection

Control HTTP Headers

• Prevent Caching 100%
• Remove IE imagetoolbar without meta tag
• Add Privacy (P3P) Header to your site
• Add language and charset headers without meta tags

Blocking Spam and bad Bots

Want to block a bad robot or web scraper using .htaccess files? Here are 2 methods that illustrate blocking 436 various user-agents. You can block them using either SetEnvIf methods, or by using Rewrite Blocks.

PHP htaccess tips

By using some cool .htaccess tricks we can control PHP to be run as a cgi or a module. If php is run as a cgi then we need to compile it ourselves or use .htaccess to force php to use a local php.ini file. If it is running as a module then we can use various directives supplied by that modules in .htaccess

• When php run as CGI
• Use a custom php.ini with mod_php or php as a cgi
• When php run as Apache Module (mod_php)
• When cgi php is run with wrapper (FastCGI)

HTTP to HTTPS Redirects with mod_rewrite

This is freaking sweet if you use SSL I promise you! Basically instead of having to check for HTTPS using a RewriteCond %{HTTPS} =on for every redirect that can be either HTTP or HTTPS, I set an environment variable once with the value "http" or "https" if HTTP or HTTPS is being used for that request, and use that env variable in the RewriteRule.

SSL in .htaccess

• Redirect non-https requests to https server
• Rewrite non-https to HTTPS without mod_ssl!
• Redirect everything served on port 80 to HTTPS URI

SetEnvIf and SetEnvIfNoCase in .htaccess

• Unique mod_setenvif Variables
• Populates HTTP_MY_ Variables with mod_setenvif variable values
• Allows only if HOST Header is present in request
• Add values from HTTP Headers

Site Security with .htaccess

chmod .htpasswd files 640, chmod .htaccess 644, php files 600, and chmod files that you really dont want people to see as 400. (NEVER chmod 777, try 766)

• CHMOD your files
• Prevent access to .htaccess and .htpasswd files
• Show Source Code instead of executing
• Securing directories: Remove ability to execute scripts
• .htaccess ErrorDocuments

.htaccess Security with MOD_SECURITY

• mod_security + mod_rewrite
• Block Spam by examining POST form fields
• Disabling mod_security conditionally per IP
• Disabling mod_security with .htaccess Authorization
• Block WordPress Spam Forever!
• Force Any Connections to be Paused a set number of ms
• ModSecurity Debugging and Logging
• Turn Off/On Logging JUST for your IP Address
• Mod_Security Directives for DreamHost
• Example httpd.conf mod_security rule files

Merging Notes

The order of merging is:

1. <Directory> (except regular expressions) and .htaccess done simultaneously (with .htaccess, if allowed, overriding <Directory>)
2. <DirectoryMatch> (and <Directory ~>)
3. <Files> and <FilesMatch> done simultaneously
4. <Location> and <LocationMatch> done simultaneously

Below is an artificial example to show the order of merging. Assuming they all apply to the request, the directives in this example will be applied in the order:

A > B > C > D > E

<Location />
E
</Location>
<Files askapache.txt>
D
</Files>
<VirtualHost *>
<Directory /a/b>
B
</Directory>
</VirtualHost>
<DirectoryMatch "^.*b$">
C
</DirectoryMatch>
<Directory /a/b>
A
</Directory>

My Favorite .htaccess Links

These are just some of my favorite .htaccess resources. I'm really into doing your own hacking to get knowledge and these links are all great resources in that respect. I'm really interested in new or unusual htaccess solutions or htaccess hacks using .htaccess files, so let me know if you find one.

NCSA HTTPd Tutorials

Robert Hansen
Here's a great Hardening HTAccess part 1, part 2, part 3 article that goes into detail about some of the rarer security applications for .htaccess files.

SAMAXES
Some very detailed and helpful .htaccess articles, such as the ".htaccess - gzip and cache your site for faster loading and bandwidth saving."

PerishablePress
Stupid .htaccess tricks is probably the best explanation online for many of the best .htaccess solutions, including many from this page. Unlike me they are fantastic writers, even for technical stuff they are very readable, so its a good blog to kick back on and read. They also have a fantastic article detailing how to block/deny specific requests using mod_rewrite.

BlogSecurity
Mostly a site for... blog security (which is really any web-app security) this blog has a few really impressive articles full of solid information for Hardening WordPress with .htaccess among more advanced topics that can be challenging but effective. This is a good site to subscribe to their feed, they publish plugin exploits and wordpress core vulnerabilities quite a bit.

Check-These
Oldschool security/unix dude with some incredibly detailed mod_rewrite tutorials, helped me the most when I first got into this, and a great guy too. See: Basic Mod_Rewrite Guide, and Advanced Mod_Rewrite Tutorial

Reaper-X
Alot of .htaccess tutorials and code. See: Hardening WordPress with Mod Rewrite and htaccess

jdMorgan
jdMorgan is the Moderator of the Apache Forum at WebmasterWorld, a great place for answers. In my experience he can answer any tough question pertaining to advanced .htaccess usage, haven't seen him stumped yet.

The W3C
Setting Charset in .htaccess is very informative.

Holy Shmoly!
A great blogger with analysis of attacks and spam. See: More ways to stop spammers and unwanted traffic.

Apache Week
A partnership with Red Hat back in the 90's that produced some excellent documentation.

Corz
Here's a resource that I consider to have some of the most creative and ingenious ideas for .htaccess files, although the author is somewhat of a character ;) Its a trip trying to navigate around the site, a fun trip. Its like nothing I've ever seen. There are only a few articles on the site, but the htaccess articles are very original and well-worth a look. See: htaccess tricks and tips.

Htaccess Directives

This is an AskApache.com exclusive you won't find this anywhere else.

Directory, DirectoryMatch, Files, FilesMatch, IfDefine, IfVersion, IfModule, Limit, LimitExcept, Location, LocationMatch, Proxy, ProxyMatch, VirtualHost, AcceptMutex, AcceptPathInfo, AccessFileName, Action, AddCharset, AddDefaultCharset, AddDescription, AddEncoding, AddHandler, AddInputFilter, AddLanguage, AddOutputFilter, AddOutputFilterByType, AddType, Alias, AliasMatch, AllowCONNECT, AllowOverride, Anonymous, Anonymous_Authoritative, Anonymous_LogEmail, Anonymous_MustGiveEmail, Anonymous_NoUserId, Anonymous_VerifyEmail, AuthAuthoritative, AuthDBMAuthoritative, AuthDBMGroupFile, AuthDBMType, AuthDBMUserFile, AuthDigestAlgorithm, AuthDigestDomain, AuthDigestFile, AuthDigestGroupFile, AuthDigestNcCheck, AuthDigestNonceFormat, AuthDigestNonceLifetime, AuthDigestQop, AuthDigestShmemSize, AuthGroupFile, AuthName, AuthType, AuthUserFile, BS2000Account, BrowserMatch, BrowserMatchNoCase, CacheNegotiatedDocs, CharsetDefault, CharsetOptions, CharsetSourceEnc, CheckSpelling, ContentDigest, CookieDomain, CookieExpires, CookieName, CookieStyle, CookieTracking, CoreDumpDirectory, DAV, DAVDepthInfinity, DAVMinTimeout, DefaultIcon, DefaultLanguage, DefaultType, DocumentRoot, ErrorDocument, ErrorLog, ExtFilterDefine, ExtFilterOptions, FancyIndexing, FileETag, ForceLanguagePriority, ForceType, GprofDir, Header, HeaderName, HostnameLookups, IdentityCheck, ImapBase, ImapDefault, ImapMenu, Include, IndexIgnore, LanguagePriority, LimitRequestBody, LimitRequestFields, LimitRequestFieldsize, LimitRequestLine, LimitXMLRequestBody, LockFile, LogLevel, MaxRequestsPerChild, MultiviewsMatch, NameVirtualHost, NoProxy, Options, PassEnv, PidFile, Port, ProxyBlock, ProxyDomain, ProxyErrorOverride, ProxyIOBufferSize, ProxyMaxForwards, ProxyPass, ProxyPassReverse, ProxyPreserveHost, ProxyReceiveBufferSize, ProxyRemote, ProxyRemoteMatch, ProxyRequests, ProxyTimeout, ProxyVia, RLimitCPU, RLimitMEM, RLimitNPROC, ReadmeName, Redirect, RedirectMatch, RedirectPermanent, RedirectTemp, RemoveCharset, RemoveEncoding, RemoveHandler, RemoveInputFilter, RemoveLanguage, RemoveOutputFilter, RemoveType, RequestHeader, Require, RewriteCond, RewriteRule, SSIEndTag, SSIErrorMsg, SSIStartTag, SSITimeFormat, SSIUndefinedEcho, Satisfy, ScoreBoardFile, Script, ScriptAlias, ScriptAliasMatch, ScriptInterpreterSource, ServerAdmin, ServerAlias, ServerName, ServerPath, ServerRoot, ServerSignature, ServerTokens, SetEnv, SetEnvIf, SetEnvIfNoCase, SetHandler, SetInputFilter, SetOutputFilter, Timeout, TypesConfig, UnsetEnv, UseCanonicalName, XBitHack, allow, deny, order, CGIMapExtension, EnableMMAP, ISAPIAppendLogToErrors, ISAPIAppendLogToQuery, ISAPICacheFile, ISAPIFakeAsync, ISAPILogNotSupported, ISAPIReadAheadBuffer, SSLLog, SSLLogLevel, MaxMemFree, ModMimeUsePathInfo, EnableSendfile, ProxyBadHeader, AllowEncodedSlashes, LimitInternalRecursion, EnableExceptionHook, TraceEnable, ProxyFtpDirCharset, AuthBasicAuthoritative, AuthBasicProvider, AuthDefaultAuthoritative, AuthDigestProvider, AuthLDAPAuthzEnabled, AuthLDAPBindDN, AuthLDAPBindPassword, AuthLDAPCharsetConfig, AuthLDAPCompareDNOnServer, AuthLDAPDereferenceAliases, AuthLDAPGroupAttribute, AuthLDAPGroupAttributeIsDN, AuthLDAPRemoteUserIsDN, AuthLDAPURL, AuthzDBMAuthoritative, AuthzDBMType, AuthzDefaultAuthoritative, AuthzGroupFileAuthoritative, AuthzLDAPAuthoritative, AuthzOwnerAuthoritative, AuthzUserAuthoritative, BalancerMember, DAVGenericLockDB, FilterChain, FilterDeclare, FilterProtocol, FilterProvider, FilterTrace, IdentityCheckTimeout, IndexStyleSheet, ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath, ProxySet, ProxyStatus, ThreadStackSize, AcceptFilter, Protocol, AuthDBDUserPWQuery, AuthDBDUserRealmQuery, UseCanonicalPhysicalPort, CheckCaseOnly, AuthLDAPRemoteUserAttribute, ProxyPassMatch, SSIAccessEnable, Substitute, ProxyPassInterpolateEnv

Htaccess Modules

Here are most of the modules that come with Apache. Each one can have new commands that can be used in .htaccess file scopes.

mod_actions, mod_alias, mod_asis, mod_auth_basic, mod_auth_digest, mod_authn_anon, mod_authn_dbd, mod_authn_dbm, mod_authn_default, mod_authn_file, mod_authz_dbm, mod_authz_default, mod_authz_groupfile, mod_authz_host, mod_authz_owner, mod_authz_user, mod_autoindex, mod_cache, mod_cern_meta, mod_cgi, mod_dav, mod_dav_fs, mod_dbd, mod_deflate, mod_dir, mod_disk_cache, mod_dumpio, mod_env, mod_expires, mod_ext_filter, mod_file_cache, mod_filter, mod_headers, mod_ident, mod_imagemap, mod_include, mod_info, mod_log_config, mod_log_forensic, mod_logio, mod_mem_cache, mod_mime, mod_mime_magic, mod_negotiation, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_ftp, mod_proxy_http, mod_rewrite, mod_setenvif, mod_speling, mod_ssl, mod_status, mod_substitute, mod_unique_id, mod_userdir, mod_usertrack, mod_version, mod_vhost_alias

Htaccess Software

Apache HTTP Server comes with the following programs.

httpd

Apache hypertext transfer protocol server

apachectl

Apache HTTP server control interface

ab

Apache HTTP server benchmarking tool

apxs

APache eXtenSion tool

dbmmanage

Create and update user authentication files in DBM format for basic authentication

fcgistarter

Start a FastCGI program

htcacheclean

Clean up the disk cache

htdigest

Create and update user authentication files for digest authentication

htdbm

Manipulate DBM password databases.

htpasswd

Create and update user authentication files for basic authentication

httxt2dbm

Create dbm files for use with RewriteMap

logresolve

Resolve hostnames for IP-addresses in Apache logfiles

log_server_status

Periodically log the server's status

rotatelogs

Rotate Apache logs without having to kill the server

split-logfile

Split a multi-vhost logfile into per-host logfiles

suexec

Switch User For Exec

Technical Look at .htaccess

Source: Apache API notes

Per-directory configuration structures

Let's look out how all of this plays out in mod_mime.c, which defines the file typing handler which emulates the NCSA server's behavior of determining file types from suffixes. What we'll be looking at, here, is the code which implements the AddType and AddEncoding commands. These commands can appear in .htaccess files, so they must be handled in the module's private per-directory data, which in fact, consists of two separate tables for MIME types and encoding information, and is declared as follows:

table *forced_types;      /* Additional AddTyped stuff */
table *encoding_types;    /* Added with AddEncoding... */
mime_dir_config;

When the server is reading a configuration file, or <Directory> section, which includes one of the MIME module's commands, it needs to create a mime_dir_config structure, so those commands have something to act on. It does this by invoking the function it finds in the module's `create per-dir config slot', with two arguments: the name of the directory to which this configuration information applies (or NULL for srm.conf), and a pointer to a resource pool in which the allocation should happen.

(If we are reading a .htaccess file, that resource pool is the per-request resource pool for the request; otherwise it is a resource pool which is used for configuration data, and cleared on restarts. Either way, it is important for the structure being created to vanish when the pool is cleared, by registering a cleanup on the pool if necessary).

For the MIME module, the per-dir config creation function just ap_pallocs the structure above, and a creates a couple of tables to fill it. That looks like this:

void *create_mime_dir_config (pool *p, char *dummy)
mime_dir_config *new = (mime_dir_config *) ap_palloc (p, sizeof(mime_dir_config));
 
new->forced_types = ap_make_table (p, 4);
new->encoding_types = ap_make_table (p, 4);

Now, suppose we've just read in a .htaccess file. We already have the per-directory configuration structure for the next directory up in the hierarchy. If the .htaccess file we just read in didn't have any AddType or AddEncoding commands, its per-directory config structure for the MIME module is still valid, and we can just use it. Otherwise, we need to merge the two structures somehow.

To do that, the server invokes the module's per-directory config merge function, if one is present. That function takes three arguments: the two structures being merged, and a resource pool in which to allocate the result. For the MIME module, all that needs to be done is overlay the tables from the new per-directory config structure with those from the parent:

void *merge_mime_dir_configs (pool *p, void *parent_dirv, void *subdirv)
mime_dir_config *parent_dir = (mime_dir_config *)parent_dirv;
mime_dir_config *subdir = (mime_dir_config *)subdirv;
mime_dir_config *new =  (mime_dir_config *)ap_palloc (p, sizeof(mime_dir_config));
new->forced_types = ap_overlay_tables (p, subdir->forced_types, parent_dir->forced_types);
new->encoding_types = ap_overlay_tables (p, subdir->encoding_types, parent_dir->encoding_types);

As a note --- if there is no per-directory merge function present, the server will just use the subdirectory's configuration info, and ignore the parent's. For some modules, that works just fine (e.g., for the includes module, whose per-directory configuration information consists solely of the state of the XBITHACK), and for those modules, you can just not declare one, and leave the corresponding structure slot in the module itself NULL.

Command handling

Now that we have these structures, we need to be able to figure out how to fill them. That involves processing the actual AddType and AddEncoding commands. To find commands, the server looks in the module's command table. That table contains information on how many arguments the commands take, and in what formats, where it is permitted, and so forth. That information is sufficient to allow the server to invoke most command-handling functions with pre-parsed arguments. Without further ado, let's look at the AddType command handler, which looks like this (the AddEncoding command looks basically the same, and won't be shown here):

char *add_type(cmd_parms *cmd, mime_dir_config *m, char *ct, char *ext)
if (*ext == '.') ++ext;
ap_table_set (m->forced_types, ext, ct);

This command handler is unusually simple. As you can see, it takes four arguments, two of which are pre-parsed arguments, the third being the per-directory configuration structure for the module in question, and the fourth being a pointer to a cmd_parms structure. That structure contains a bunch of arguments which are frequently of use to some, but not all, commands, including a resource pool (from which memory can be allocated, and to which cleanups should be tied), and the (virtual) server being configured, from which the module's per-server configuration data can be obtained if required.

Another way in which this particular command handler is unusually simple is that there are no error conditions which it can encounter. If there were, it could return an error message instead of NULL; this causes an error to be printed out on the server's stderr, followed by a quick exit, if it is in the main config files; for a .htaccess file, the syntax error is logged in the server error log (along with an indication of where it came from), and the request is bounced with a server error response (HTTP error status, code 500).

The MIME module's command table has entries for these commands, which look like this:

command_rec mime_cmds[] =
{ "AddType", add_type, NULL, OR_FILEINFO, TAKE2, "a mime type followed by a file extension" },
{ "AddEncoding", add_encoding, NULL, OR_FILEINFO, TAKE2, "an encoding (e.g., gzip), followed by a file extension" },

The entries in these tables are:

• The name of the command
• The function which handles it a (void *) pointer, which is passed in the cmd_parms structure to the command handler --- this is useful in case many similar commands are handled by the same function.
• A bit mask indicating where the command may appear. There are mask bits corresponding to each AllowOverride option, and an additional mask bit, RSRC_CONF, indicating that the command may appear in the server's own config files, but not in any .htaccess file.
• A flag indicating how many arguments the command handler wants pre-parsed, and how they should be passed in. TAKE2 indicates two pre-parsed arguments. Other options are TAKE1, which indicates one pre-parsed argument, FLAG, which indicates that the argument should be On or Off, and is passed in as a boolean flag, RAW_ARGS, which causes the server to give the command the raw, unparsed arguments (everything but the command name itself). There is also ITERATE, which means that the handler looks the same as TAKE1, but that if multiple arguments are present, it should be called multiple times, and finally ITERATE2, which indicates that the command handler looks like a TAKE2, but if more arguments are present, then it should be called multiple times, holding the first argument constant.
• Finally, we have a string which describes the arguments that should be present. If the arguments in the actual config file are not as required, this string will be used to help give a more specific error message. (You can safely leave this NULL).

Finally, having set this all up, we have to use it. This is ultimately done in the module's handlers, specifically for its file-typing handler, which looks more or less like this; note that the per-directory configuration structure is extracted from the request_rec's per-directory configuration vector by using the ap_get_module_config function.

Side notes --- per-server configuration, virtual servers, etc.

The basic ideas behind per-server module configuration are basically the same as those for per-directory configuration; there is a creation function and a merge function, the latter being invoked where a virtual server has partially overridden the base server configuration, and a combined structure must be computed. (As with per-directory configuration, the default if no merge function is specified, and a module is configured in some virtual server, is that the base configuration is simply ignored).

The only substantial difference is that when a command needs to configure the per-server private module data, it needs to go to the cmd_parms data to get at it. Here's an example, from the alias module, which also indicates how a syntax error can be returned (note that the per-directory configuration argument to the command handler is declared as a dummy, since the module doesn't actually have per-directory config data):

Continue Reading Page 2

THE Ultimate Htaccess originally appeared on AskApache.com

PHP's function fsockopen lets you open an Internet or Unix domain socket connection for connecting to a resource, and is one of the most powerful functions. fsockopen could be described as creating a direct link to the wire connected to a resource, which means you can send any information (EBCDIC, ASCII, Hex, C arrays, Raw) directly to the target server.

A Socket is like /dev/null

In unix you can send anything to the /dev/null device, for Windows think Recycle Bin, and likewise you can send anything to a socket created with fsockopen. I've seen fsockopen code that sends custom exploits to cisco routers, including being used by the metasploit framework. I've seen fsockopen telnet emulation, smtp/pop3 login, and a lot of other advanced raw networking that is exciting for me see.

Some Definitions for Fsockopen

client

A program that establishes connections for the purpose of sending requests.

server

An application program that accepts connections in order to service requests by sending back responses.

Simple Socket Explantion

A web server host listens on TCP port 80. When a client host wishes to view a resource on the web server, it establishes a TCP connection with the server host by opening a socket to send the request for the resource. When the connection is established, the client and server exchange requests and responses (respectively) until the connection is closed or aborted.

HTTP and fsockopen

The Snoopy class is bundled with WordPress distributions and uses fsockopen to achieve most of its cool features. WordPress core, plugins, and other included files and classes also use the fsockopen function to communicate via HTTP.

Fsockopen Examples

Note the warning sign, fsockopen is dangerous in the sense that you can crash your server, perform a DOS against your own server or other site, use up all your servers available sockets and fd descriptors, use up your bandwidth, etc.. Shouldn't be a problem unless you are being malicious or careless.

Here are some BOSS fsockopen functions I hacked together yesterday for use in my AskApache Crazy Cache WordPress Plugin. I've used code and ideas from 100's of authors, projects, and docs to try to make this the very best I can.

Intro

This is a working example employing as many of the best-practices, tips, and tricks for using fsockopen on remote streams that I could find.

<?php
// max time for script execution
if(!@defined('AA_MAX_TIME')) define('AA_MAX_TIME',  60);
 
// max time for socket reads
if(!@defined('AA_RECV_TIME')) define('AA_RECV_TIME', 30);
 
// max time for socket connect
if(!@defined('AA_CONN_TIME')) define('AA_CONN_TIME', 5);
 
// linebreak
if(!@defined('AA_LF')) define('AA_LF', chr(13).chr(10));
 
// ignore TCP RST i.e. browser stop button
@ignore_user_abort(1);
 
// set the script execution time
@set_time_limit(AA_MAX_TIME);
 
// set the default socket timeout value
@ini_set("default_socket_timeout",AA_RECV_TIME);
 
// output implicitly
@ob_implicit_flush(1);
 
// for binary freads
@set_magic_quotes_runtime(0);
 
// keep track of script execution time
$aa_time=time();
 
// download each of these urls using fsockopen
aa_dl('http://httpd.apache.org');
aa_dl('http://www.w3.org');
aa_dl('http://www.google.com');
aa_dl('http://www.freebsd.org/cgi/man.cgi?query=connect&sektion=2&apropos=0&manpath=FreeBSD+7.0-RELEASE');
aa_dl('http://www.askapache.com/htaccess/htaccess.html');
aa_dl('http://www.php.net');
aa_dl('http://en.wikipedia.org/wiki/Main_Page');
 
/*  returns a socket pointer if valid or displays an error message
    sets stream timeout, starts the clock to check for socket read time */
function askapache_get_sock($target,$port){
  global $aa_time_start;
  $aa_time_start=time();
  if(false===($fp = @fsockopen($target,$port,$errno,$errstr,AA_CONN_TIME))||!is_resource($fp))
    return askapache_sock_strerror($errno,$errstr);
  @stream_set_timeout($fp, AA_RECV_TIME);
  return $fp;
}
 
/*  writes request, then reads response until EOF, script max, or socket max
    returns response on success.  Uses buffer to allow size>100megs */
function askapache_txrx($fp,$request,$chunk=1024){
  $rec=$buf='';
  if(!@fwrite($fp, $request, strlen($request)))die('fwrite error');
  while ( !@feof($fp) && askapache_time_ok(askapache_time_passed())){
    $buf = @fread($fp, $chunk);
    $rec .= $buf;
  }
  if(!@fclose($fp))die('fclose error');
  return $rec;
}
 
/* initiates the socket and download for the passed url.
   automatically handles gzip, chunked, both, and plain downloads.
   uses the long2ip/ip2long for ip validation, uses gethostbyname to
   get the ipv4 address which saves fsockopen from having to do the lookup
   final data is saved to $rbody but currently only displays headers.*/
function aa_dl($url=NULL){
  global $aa_time;
  $ub = @parse_url($url);
  if(!isset($ub['host'])||empty($ub['host'])) die("bad url $url");
  $proto   = ($ub['scheme']=='https')?'ssl://':'';
  $port   = (isset($ub['port'])&&!empty($ub['port'])) ? $ub['port']:($proto!='')?443:80;
  $path   = (isset($ub['path'])&&!empty($ub['path'])) ? $ub['path']:'/';
  $query   = (isset($ub['query'])&&!empty($ub['query'])) ? '?'.$ub['query'] : '';
  $host   = $ub['host'];
  $ipp     = @gethostbyname($host);
  $ip     = ($ipp!=$host) ? long2ip(ip2long($ipp)) : $host;
 
  $headers=array(
   "GET {$path}{$query} HTTP/1.1",
   "Host: {$host}",
   'User-Agent: Mozilla/5.0 (AskApache/; +http://www.askapache.com/)',
   'Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,*/*;q=0.5',
   'Accept-Language: en-us,en;q=0.5',
   'Accept-Encoding: gzip,deflate',
   'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
   'Connection: close','Referer: http://www.askapache.com'
  );
  $request=join(AA_LF,$headers).AA_LF.AA_LF;
 
  $fp=askapache_get_sock($proto.$ip, $port);
  if($fp){
    $rbody=$rec='';$resp_headers=array();
    $rec=askapache_txrx($fp,$request);
    list($resp_headers, $rbody) = explode(AA_LF.AA_LF, trim($rec), 2);
  echo "\n<p>$request</p>\n<p>$resp_headers</p>\n";
    $gzip2=(stripos($resp_headers,'Content-Encoding')!==false &&
        stripos($resp_headers,'gzip')!==false)?1:0;
    $chunk=(stripos($resp_headers,'Transfer-Encoding')!==false &&
        stripos($resp_headers,'chunked')!==false)?1:0;
    $rbody=aa_decode_body($rbody,$chunk,$gzip2);
    unset($rbody);
  }
}
 
/* based on http://us.php.net/manual/en/function.fsockopen.php#75175
   ungzips and/or re-assembles transfer-encoded:chunked responses
   returns the good response on success */
function aa_decode_body ($str, $chunked, $gzipped){
  if($gzipped && !$chunked) return aa_gzdecode($str);
  if(!$gzipped && !$chunked) return $str;
  $tmp = $str; $str = '';
  do {
    $tmp = ltrim($tmp);
    $pos = strpos($tmp,AA_LF);
    $len = hexdec(substr($tmp, 0, $pos));
    if($gzipped) $str .= gzinflate(substr($tmp,($pos+12),$len));
    else $str .=substr($tmp,($pos+2),$len);
    $tmp = substr($tmp,($len+$pos+2));
  $chk=trim($tmp);
  } while (!empty($chk));
  return $str;
}
 
/*  based on http://us2.php.net/manual/en/function.gzencode.php#82520
  saves the gzipped data to a tempfile, then outputs the decoded
  data to the output buffer using readgzfile, returning the decoded
  buffer and deleting the tempfile on success */
function aa_gzdecode($data){
  $g=tempnam('/tmp','ff');
  @file_put_contents($g,$data);
  ob_start(); readgzfile($g); $d=ob_get_clean(); @unlink($g);
  return $d;
}
 
/*  very cool!  this is run during socket reads and checks whether the script
  execution time limit or the socket read time limit has been met, killing
  the script if so, otherwise returns true.  Run with a cron-like process */
function askapache_time_ok($sock_time=0) {
  global $aa_time;
  if (time()-$aa_time>AA_MAX_TIME)
    die('killed script.. time exceeded '.AA_MAX_TIME.' Total: '.$total);
  if ($sock_time>AA_RECV_TIME)
    die('Killed socket.. time exceeded '.AA_RECV_TIME.' Total: '.$sock_time);
  return true;
}
 
/* input for askapache_time_ok to keep track of each socket read time time. */
function askapache_time_passed() {
  global $aa_time_start;
  return (time() - $aa_time_start);
}
 
/*  handles fsockopen errors, printing them out though you may want to die on err */
function askapache_sock_strerror($errno,$errstr){
  switch($errno){
    case -3:  $err="Socket creation failed"; break;
    case -4:  $err="DNS lookup failure"; break;
    case -5:  $err="Connection refused or timed out"; break;
    case 111: $err="Connection refused"; break;
    case 113: $err="No route to host"; break;
    case 110: $err="Connection timed out"; break;
    case 104: $err="Connection reset by client"; break;
    default:  $err="Connection failed"; break;
  }
  echo '<p>Fsockopen failed!'."\n[".$errno."] ".$err." (".$errstr.")</p>";
  return false;
}
?>

Debugging Fsockopen

If you really want to know more about fsockopen, you can do what I did and read all the relevant php source files, your OS sys, lib, and user files relevant to fsockopen, and of course you can always trace php using the fsockopen function to get an under-the-hood look at what in the world fsockopen is doing. Personally, I was trying to find more error codes and error strings to display when an fsockopen call failed, and I ended up finding over 50..

fsockopen Errors

function fsockopen_err($errnum)
{
   static $fsockopen_errors;
   is_null($fsockopen_errors) && $fsockopen_errors = array(
  0 => 'Success',
  1 => 'Operation not permitted',
  2 => 'No such file or directory',
  3 => 'No such process',
  4 => 'Interrupted system call - DNS lookup failure',
  5 => 'Input/output error - Connection refused or timed out',
  6 => 'No such device or address',
  7 => 'Argument list too long',
  8 => 'Exec format error',
  9 => 'Bad file descriptor',
  10 => 'No child processes',
  11 => 'Resource temporarily unavailable',
  12 => 'Cannot allocate memory',
  13 => 'Permission denied',
  14 => 'Bad address',
  15 => 'Block device required',
  16 => 'Device or resource busy',
  17 => 'File exists',
  18 => 'Invalid cross-device link',
  19 => 'No such device',
  20 => 'Not a directory',
  21 => 'Is a directory',
  22 => 'Invalid argument',
  23 => 'Too many open files in system',
  24 => 'Too many open files',
  25 => 'Inappropriate ioctl for device',
  26 => 'Text file busy',
  27 => 'File too large',
  28 => 'No space left on device',
  29 => 'Illegal seek',
  30 => 'Read-only file system',
  31 => 'Too many links',
  32 => 'Broken pipe',
  33 => 'Numerical argument out of domain',
  34 => 'Numerical result out of range',
  35 => 'Resource deadlock avoided',
  36 => 'File name too long',
  37 => 'No locks available',
  38 => 'Function not implemented',
  39 => 'Directory not empty',
  40 => 'Too many levels of symbolic links',
  41 => 'Unknown error 41',
  42 => 'No message of desired type',
  43 => 'Identifier removed',
  44 => 'Channel number out of range',
  45 => 'Level 2 not synchronized',
  46 => 'Level 3 halted',
  47 => 'Level 3 reset',
  48 => 'Link number out of range',
  49 => 'Protocol driver not attached',
  50 => 'No CSI structure available',
  51 => 'Level 2 halted',
  52 => 'Invalid exchange',
  53 => 'Invalid request descriptor',
  54 => 'Exchange full',
  55 => 'No anode',
  56 => 'Invalid request code',
  57 => 'Invalid slot',
  58 => 'Unknown error 58',
  59 => 'Bad font file format',
  60 => 'Device not a stream',
  61 => 'No data available',
  62 => 'Timer expired',
  63 => 'Out of streams resources',
  64 => 'Machine is not on the network',
  65 => 'Package not installed',
  66 => 'Object is remote',
  67 => 'Link has been severed',
  68 => 'Advertise error',
  69 => 'Srmount error',
  70 => 'Communication error on send',
  71 => 'Protocol error',
  72 => 'Multihop attempted',
  73 => 'RFS specific error',
  74 => 'Bad message',
  75 => 'Value too large for defined data type',
  76 => 'Name not unique on network',
  77 => 'File descriptor in bad state',
  78 => 'Remote address changed',
  79 => 'Can not access a needed shared library',
  80 => 'Accessing a corrupted shared library',
  81 => '.lib section in a.out corrupted',
  82 => 'Attempting to link in too many shared libraries',
  83 => 'Cannot exec a shared library directly',
  84 => 'Invalid or incomplete multibyte or wide character',
  85 => 'Interrupted system call should be restarted',
  86 => 'Streams pipe error',
  87 => 'Too many users',
  88 => 'Socket operation on non-socket',
  89 => 'Destination address required',
  90 => 'Message too long',
  91 => 'Protocol wrong type for socket',
  92 => 'Protocol not available',
  93 => 'Protocol not supported',
  94 => 'Socket type not supported',
  95 => 'Operation not supported',
  96 => 'Protocol family not supported',
  97 => 'Address family not supported by protocol',
  98 => 'Address already in use',
  99 => 'Cannot assign requested address',
  100 => 'Network is down',
  101 => 'Network is unreachable',
  102 => 'Network dropped connection on reset',
  103 => 'Software caused connection abort',
  104 => 'Connection reset by peer',
  105 => 'No buffer space available',
  106 => 'Transport endpoint is already connected',
  107 => 'Transport endpoint is not connected',
  108 => 'Cannot send after transport endpoint shutdown',
  109 => 'Too many references: cannot splice',
  110 => 'Connection timed out',
  111 => 'Connection refused',
  112 => 'Host is down',
  113 => 'No route to host',
  114 => 'Operation already in progress',
  115 => 'Operation now in progress',
  116 => 'Stale NFS file handle',
  117 => 'Structure needs cleaning',
  118 => 'Not a XENIX named type file',
  119 => 'No XENIX semaphores available',
  120 => 'Is a named type file',
  121 => 'Remote I/O error',
  122 => 'Disk quota exceeded',
  123 => 'No medium found',
  124 => 'Wrong medium type',
  125 => 'Operation canceled'
  );
    return (isset($fsockopen_errors[$errnum])) ? $fsockopen_errors[$errnum] : $errnum;
}

If you would like to see all the errors on your particular machine:

for($i=0, $s=""; $i<250; $s=socket_strerror($i), $i++)
  !empty($s) && ('Unknown error' != (substr($s,0,13)) ) && print "{$i} => {$s}\n";

Which outputs:

1 => Success
2 => Operation not permitted
3 => No such file or directory
4 => No such process
5 => Interrupted system call
6 => Input/output error
7 => No such device or address
8 => Argument list too long
9 => Exec format error
10 => Bad file descriptor
11 => No child processes
12 => Resource temporarily unavailable
13 => Cannot allocate memory
14 => Permission denied
15 => Bad address
16 => Block device required
17 => Device or resource busy
18 => File exists
19 => Invalid cross-device link
20 => No such device
21 => Not a directory
22 => Is a directory
23 => Invalid argument
24 => Too many open files in system
25 => Too many open files
26 => Inappropriate ioctl for device
27 => Text file busy
28 => File too large
29 => No space left on device
30 => Illegal seek
31 => Read-only file system
32 => Too many links
33 => Broken pipe
34 => Numerical argument out of domain
35 => Numerical result out of range
36 => Resource deadlock avoided
37 => File name too long
38 => No locks available
39 => Function not implemented
40 => Directory not empty
41 => Too many levels of symbolic links
43 => No message of desired type
44 => Identifier removed
45 => Channel number out of range
46 => Level 2 not synchronized
47 => Level 3 halted
48 => Level 3 reset
49 => Link number out of range
50 => Protocol driver not attached
51 => No CSI structure available
52 => Level 2 halted
53 => Invalid exchange
54 => Invalid request descriptor
55 => Exchange full
56 => No anode
57 => Invalid request code
58 => Invalid slot
60 => Bad font file format
61 => Device not a stream
62 => No data available
63 => Timer expired
64 => Out of streams resources
65 => Machine is not on the network
66 => Package not installed
67 => Object is remote
68 => Link has been severed
69 => Advertise error
70 => Srmount error
71 => Communication error on send
72 => Protocol error
73 => Multihop attempted
74 => RFS specific error
75 => Bad message
76 => Value too large for defined data type
77 => Name not unique on network
78 => File descriptor in bad state
79 => Remote address changed
80 => Can not access a needed shared library
81 => Accessing a corrupted shared library
82 => .lib section in a.out corrupted
83 => Attempting to link in too many shared libraries
84 => Cannot exec a shared library directly
85 => Invalid or incomplete multibyte or wide character
86 => Interrupted system call should be restarted
87 => Streams pipe error
88 => Too many users
89 => Socket operation on non-socket
90 => Destination address required
91 => Message too long
92 => Protocol wrong type for socket
93 => Protocol not available
94 => Protocol not supported
95 => Socket type not supported
96 => Operation not supported
97 => Protocol family not supported
98 => Address family not supported by protocol
99 => Address already in use
100 => Cannot assign requested address
101 => Network is down
102 => Network is unreachable
103 => Network dropped connection on reset
104 => Software caused connection abort
105 => Connection reset by peer
106 => No buffer space available
107 => Transport endpoint is already connected
108 => Transport endpoint is not connected
109 => Cannot send after transport endpoint shutdown
110 => Too many references: cannot splice
111 => Connection timed out
112 => Connection refused
113 => Host is down
114 => No route to host
115 => Operation already in progress
116 => Operation now in progress
117 => Stale NFS file handle
118 => Structure needs cleaning
119 => Not a XENIX named type file
120 => No XENIX semaphores available
121 => Is a named type file
122 => Remote I/O error
123 => Disk quota exceeded
124 => No medium found
125 => Wrong medium type
126 => Operation canceled

Tracing fsockopen using Strace

Once you save the above file on your site, you can use the strace tool to debug it. This is a tad overboard but way cool nevertheless!

strace -e trace=connect php -nef fsockopen-test.php

connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.33.216.129")}, 28) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.87.106.226")}, 16) = -1 EINPROGRESS (Operation now in progress)

strace -e trace=network php -nef fsockopen-test.php

socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.33.216.129")}, 28) = 0
send(3, "\274\221\1\0\0\1\0\0\0\0\0\0\5httpd\6apache\3org\0\0\1"..., 34, 0) = 34
recvfrom(3, "\274\221\201\200\0\1\0\1\0\0\0\0\5httpd\6apache\3org\0"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.33.216.129")}, [16]) = 50
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.87.106.226")}, 16) = -1 EINPROGRESS (Operation now in progress)
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
send(3, "GET / HTTP/1.1\r\nHost: httpd.apac"..., 356, MSG_DONTWAIT) = 356
recv(3, "HTTP/1.1 200 OK\r\nDate: Wed, 02 J"..., 8192, MSG_DONTWAIT) = 2609
recv(3, "", 8192, MSG_DONTWAIT)         = 0

strace -q -e trace=all php -nef fsockopen-test.php

mmap2(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb76ba000
munmap(0xb76ba000, 266240)              = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
close(3)                                = 0
open("/etc/hosts", O_RDONLY)            = 3
fcntl64(3, F_GETFD)                     = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=948, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f6e000
read(3, "# /etc/hosts - dh2 generated\n127"..., 4096) = 948
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0xb7f6e000, 4096)                = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.33.216.129")}, 28) = 0
send(3, "X~\1\0\0\1\0\0\0\0\0\0\2en\twikipedia\3org\0\0\1"..., 34, 0) = 34
gettimeofday({1214998196, 656179}, NULL) = 0
poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 5000) = 1
ioctl(3, FIONREAD, [100])               = 0
recvfrom(3, "X~\201\200\0\1\0\3\0\0\0\0\2en\twikipedia\3org\0\0\1"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("66.33.216.129")}, [16]) = 100
close(3)                                = 0
time(NULL)                              = 1214998196
gettimeofday({1214998196, 656754}, NULL) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
fcntl64(3, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("208.80.152.2")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLIN|POLLOUT|POLLERR|POLLHUP, revents=POLLOUT}], 1, 10000) = 1
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl64(3, F_SETFL, O_RDWR)             = 0
send(3, "GET /wiki/Main_Page HTTP/1.1\r\nHo"..., 370, MSG_DONTWAIT) = 370
poll([{fd=3, events=POLLIN|POLLPRI|POLLERR|POLLHUP}], 1, 0) = 0
time(NULL)                              = 1214998196
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 30000) = 1
recv(3, "HTTP/1.0 200 OK\r\nDate: Wed, 02 J"..., 8192, MSG_DONTWAIT) = 2896
time(NULL)                              = 1214998196
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 30000) = 1
recv(3, "\214!\337i\307\336\23w\253wy\215\26EL\227;\227\253\261"..., 8192, MSG_DONTWAIT) = 5792
time(NULL)                              = 1214998196
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 30000) = 1
recv(3, "4\201\273\214\17yI\347\257\371\373\344\332\330\227\245"..., 8192, MSG_DONTWAIT) = 7487
time(NULL)                              = 1214998197
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP, revents=POLLIN}], 1, 30000) = 1
recv(3, "", 8192, MSG_DONTWAIT)         = 0
close(3)                                = 0
write(1, "\n<pre>GET /wiki/Main_Page HTTP/1"..., 1300

More Fsockopen Info

TCP Multiplexing

RFC 793: To allow for many processes within a single Host to use TCP communication facilities simultaneously, the TCP provides a set of addresses or ports within each host. Concatenated with the network and host addresses from the internet communication layer, this forms a socket. A pair of sockets uniquely identifies each connection. That is, a socket may be simultaneously used in multiple connections.

The binding of ports to processes is handled independently by each Host. However, it proves useful to attach frequently used processes (e.g., a "logger" or timesharing service) to fixed sockets which are made known to the public. These services can then be accessed through the known addresses. Establishing and learning the port addresses of other processes may involve more dynamic mechanisms.

TCP Connections

The reliability and flow control mechanisms described above require that TCPs initialize and maintain certain status information for each data stream. The combination of this information, including sockets, sequence numbers, and window sizes, is called a connection. Each connection is uniquely specified by a pair of sockets identifying its two sides.

When two processes wish to communicate, their TCP's must first establish a connection (initialize the status information on each side). When their communication is complete, the connection is terminated or closed to free the resources for other uses.

Since connections must be established between unreliable hosts and over the unreliable internet communication system, a handshake mechanism with clock-based sequence numbers is used to avoid erroneous initialization of connections.

Fsockopen Practical Uses

• Download Web Pages, Files, etc.
• Upload a file
• Send POST data to a form
• Emulate cron
• Download plugin updates
• Scan sites for exploits
• Auto Login to Google
• Pass wp-nonces via cookie headers, and more

Transfer-Encoding

RFC 2068

19.4.6 Introduction of Transfer-Encoding
 
   HTTP/1.1 introduces the Transfer-Encoding header field (section
   14.40).  Proxies/gateways MUST remove any transfer coding prior to
   forwarding a message via a MIME-compliant protocol.
 
   A process for decoding the "chunked" transfer coding (section 3.6)
   can be represented in pseudo-code as:
 
          length := 0
          read chunk-size, chunk-ext (if any) and CRLF
          while (chunk-size > 0) {
             read chunk-data and CRLF
             append chunk-data to entity-body
             length := length + chunk-size
             read chunk-size and CRLF
          }
          read entity-header
          while (entity-header not empty) {
             append entity-header to existing header fields
             read entity-header
          }
          Content-Length := length
          Remove "chunked" from Transfer-Encoding

Socket-Related Man Pages

DESCRIPTION
This  manual  page  describes the Linux networking socket layer user interface. The BSD compatible sockets are the uniform interface between
the user process and the network protocol stacks in the kernel.  The protocol modules are  grouped  into  protocol  families  like  PF_INET,
PF_IPX, PF_PACKET and socket types like SOCK_STREAM or SOCK_DGRAM.  See socket(2) for more information on families and types.
 
SOCKET LAYER FUNCTIONS
These  functions  are  used by the user process to send or receive packets and to do other socket operations. For more information see their
respective manual pages.
 
socket(2) creates a socket, connect(2) connects a socket to a remote socket address, the bind(2) function binds a socket to a  local  socket
address,  listen(2)  tells  the socket that new connections shall be accepted, and accept(2) is used to get a new socket with a new incoming
connection.  socketpair(2) returns two connected anonymous sockets (only implemented for a few local families like PF_UNIX)
 
send(2), sendto(2), and sendmsg(2) send data over a socket, and recv(2), recvfrom(2), recvmsg(2) receive data from a  socket.   poll(2)  and
select(2)  wait  for  arriving  data  or a readiness to send data.  In addition, the standard I/O operations like write(2), writev(2), send-
file(2), read(2), and readv(2) can be used to read and write data.
 
getsockname(2) returns the local socket address and getpeername(2) returns the remote socket address.  getsockopt(2) and  setsockopt(2)  are
used to set or get socket layer or protocol options.  ioctl(2) can be used to set or read some other options.
 
close(2) is used to close a socket.  shutdown(2) closes parts of a full duplex socket connection.
 
Seeking, or calling pread(2) or pwrite(2) with a non-zero position is not supported on sockets.
 
It  is possible to do non-blocking IO on sockets by setting the O_NONBLOCK flag on a socket file descriptor using fcntl(2).  Then all opera-
tions that would block will (usually) return with EAGAIN (operation should be retried later); connect(2) will return EINPROGRESS error.  The
user can then wait for various events via poll(2) or select(2).

From the FreeBSD man page for socket(2)

Sockets of type SOCK_STREAM are full-duplex byte streams, similar to
pipes.  A stream socket must be in a connected state before any data may
be sent or received on it.  A connection to another socket is created
with a connect(2) system call.  Once connected, data may be transferred
using read(2) and write(2) calls or some variant of the send(2) and
recv(2) functions.  (Some protocol families, such as the Internet family,
support the notion of an ``implied connect'', which permits data to be
sent piggybacked onto a connect operation by using the sendto(2) system
call.)  When a session has been completed a close(2) may be performed.
Out-of-band data may also be transmitted as described in send(2) and
received as described in recv(2).
 
The communications protocols used to implement a SOCK_STREAM insure that
data is not lost or duplicated.  If a piece of data for which the peer
protocol has buffer space cannot be successfully transmitted within a
reasonable length of time, then the connection is considered broken and
calls will indicate an error with -1 returns and with ETIMEDOUT as the
specific code in the global variable errno.  The protocols optionally
keep sockets ``warm'' by forcing transmissions roughly every minute in
the absence of other activity.  An error is then indicated if no response
can be elicited on an otherwise idle connection for an extended period
(e.g. 5 minutes).  A SIGPIPE signal is raised if a process sends on a
broken stream; this causes naive processes, which do not handle the sig-
nal, to exit.

Have Fun ;)

define ('SOCKET_EPERM', 1);
define ('SOCKET_ENOENT', 2);
define ('SOCKET_EINTR', 4);
define ('SOCKET_EIO', 5);
define ('SOCKET_ENXIO', 6);
define ('SOCKET_E2BIG', 7);
define ('SOCKET_EBADF', 9);
define ('SOCKET_EAGAIN', 11);
define ('SOCKET_ENOMEM', 12);
define ('SOCKET_EACCES', 13);
define ('SOCKET_EFAULT', 14);
define ('SOCKET_ENOTBLK', 15);
define ('SOCKET_EBUSY', 16);
define ('SOCKET_EEXIST', 17);
define ('SOCKET_EXDEV', 18);
define ('SOCKET_ENODEV', 19);
define ('SOCKET_ENOTDIR', 20);
define ('SOCKET_EISDIR', 21);
define ('SOCKET_EINVAL', 22);
define ('SOCKET_ENFILE', 23);
define ('SOCKET_EMFILE', 24);
define ('SOCKET_ENOTTY', 25);
define ('SOCKET_ENOSPC', 28);
define ('SOCKET_ESPIPE', 29);
define ('SOCKET_EROFS', 30);
define ('SOCKET_EMLINK', 31);
define ('SOCKET_EPIPE', 32);
define ('SOCKET_ENAMETOOLONG', 36);
define ('SOCKET_ENOLCK', 37);
define ('SOCKET_ENOSYS', 38);
define ('SOCKET_ENOTEMPTY', 39);
define ('SOCKET_ELOOP', 40);
define ('SOCKET_EWOULDBLOCK', 11);
define ('SOCKET_ENOMSG', 42);
define ('SOCKET_EIDRM', 43);
define ('SOCKET_ECHRNG', 44);
define ('SOCKET_EL2NSYNC', 45);
define ('SOCKET_EL3HLT', 46);
define ('SOCKET_EL3RST', 47);
define ('SOCKET_ELNRNG', 48);
define ('SOCKET_EUNATCH', 49);
define ('SOCKET_ENOCSI', 50);
define ('SOCKET_EL2HLT', 51);
define ('SOCKET_EBADE', 52);
define ('SOCKET_EBADR', 53);
define ('SOCKET_EXFULL', 54);
define ('SOCKET_ENOANO', 55);
define ('SOCKET_EBADRQC', 56);
define ('SOCKET_EBADSLT', 57);
define ('SOCKET_ENOSTR', 60);
define ('SOCKET_ENODATA', 61);
define ('SOCKET_ETIME', 62);
define ('SOCKET_ENOSR', 63);
define ('SOCKET_ENONET', 64);
define ('SOCKET_EREMOTE', 66);
define ('SOCKET_ENOLINK', 67);
define ('SOCKET_EADV', 68);
define ('SOCKET_ESRMNT', 69);
define ('SOCKET_ECOMM', 70);
define ('SOCKET_EPROTO', 71);
define ('SOCKET_EMULTIHOP', 72);
define ('SOCKET_EBADMSG', 74);
define ('SOCKET_ENOTUNIQ', 76);
define ('SOCKET_EBADFD', 77);
define ('SOCKET_EREMCHG', 78);
define ('SOCKET_ERESTART', 85);
define ('SOCKET_ESTRPIPE', 86);
define ('SOCKET_EUSERS', 87);
define ('SOCKET_ENOTSOCK', 88);
define ('SOCKET_EDESTADDRREQ', 89);
define ('SOCKET_EMSGSIZE', 90);
define ('SOCKET_EPROTOTYPE', 91);
define ('SOCKET_ENOPROTOOPT', 92);
define ('SOCKET_EPROTONOSUPPORT', 93);
define ('SOCKET_ESOCKTNOSUPPORT', 94);
define ('SOCKET_EOPNOTSUPP', 95);
define ('SOCKET_EPFNOSUPPORT', 96);
define ('SOCKET_EAFNOSUPPORT', 97);
define ('SOCKET_EADDRINUSE', 98);
define ('SOCKET_EADDRNOTAVAIL', 99);
define ('SOCKET_ENETDOWN', 100);
define ('SOCKET_ENETUNREACH', 101);
define ('SOCKET_ENETRESET', 102);
define ('SOCKET_ECONNABORTED', 103);
define ('SOCKET_ECONNRESET', 104);
define ('SOCKET_ENOBUFS', 105);
define ('SOCKET_EISCONN', 106);
define ('SOCKET_ENOTCONN', 107);
define ('SOCKET_ESHUTDOWN', 108);
define ('SOCKET_ETOOMANYREFS', 109);
define ('SOCKET_ETIMEDOUT', 110);
define ('SOCKET_ECONNREFUSED', 111);
define ('SOCKET_EHOSTDOWN', 112);
define ('SOCKET_EHOSTUNREACH', 113);
define ('SOCKET_EALREADY', 114);
define ('SOCKET_EINPROGRESS', 115);
define ('SOCKET_EISNAM', 120);
define ('SOCKET_EREMOTEIO', 121);
define ('SOCKET_EDQUOT', 122);
define ('SOCKET_ENOMEDIUM', 123);
define ('SOCKET_EMEDIUMTYPE', 124);

• Hypertext Transfer Protocol — HTTP/1.1, RFC 2616. R. Fielding et al.
• Hypertext ransport Protocol HTTP/1.1. J. Gettys. (slides)
• What's wrong with HTTP (and why it doesn't matter).J. C. Mogul. (PDF slides)
• Network Performance Effects of HTTP/1.1, CSS1, and PNG.H. F. Nielsen, J. Gettys et al.
• Mozilla's HTTP/1.1 Pipelining FAQ. D. Fisher.
• Wikipedia: HTTP proxy.

Fsockopen Power Plays originally appeared on AskApache.com

This article shows 2 methods of blocking this entire list of bad robots and web scrapers with .htaccess files using SetEnvIfNoCase or using RewriteRules with mod_rewrite

Blocking Bad Robots and Web Scrapers with RewriteRules

ErrorDocument 403 /403.html
 
RewriteEngine On
RewriteBase /
 
# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww-perl|widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) [NC,OR]
 
# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC,OR]
 
# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
 
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule . - [F,L]

Alternate RewriteCond Rules

RewriteEngine on
 
#Block spambots
RewriteCond %{HTTP:User-Agent} (?:Alexibot|Art-Online|asterias|BackDoorbot|Black.Hole|\
BlackWidow|BlowFish|botALot|BuiltbotTough|Bullseye|BunnySlippers|Cegbfeieh|Cheesebot|\
CherryPicker|ChinaClaw|CopyRightCheck|cosmos|Crescent|Custo|DISCo|DittoSpyder|DownloadsDemon|\
eCatch|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|ExpresssWebPictures|ExtractorPro|\
EyeNetIE|FlashGet|Foobot|FrontPage|GetRight|GetWeb!|Go-Ahead-Got-It|Go!Zilla|GrabNet|Grafula|\
Harvest|hloader|HMView|httplib|HTTrack|humanlinks|ImagesStripper|ImagesSucker|IndysLibrary|\
InfonaviRobot|InterGET|Internet\sNinja|Jennybot|JetCar|JOC\sWeb\sSpider|Kenjin.Spider|Keyword.Density|\
larbin|LeechFTP|Lexibot|libWeb/clsHTTP|LinkextractorPro|LinkScan/8.1a.Unix|LinkWalker|lwp-trivial|\
Mass\sDownloader|Mata.Hari|Microsoft.URL|MIDown\stool|MIIxpc|Mister.PiX|Mister\sPiX|moget|\
Mozilla/3.Mozilla/2.01|Mozilla.*NEWT|Navroad|NearSite|NetAnts|NetMechanic|NetSpider|Net\sVampire|\
NetZIP|NICErsPRO|NPbot|Octopus|Offline.Explorer|Offline\sExplorer|Offline\sNavigator|Openfind|\
Pagerabber|Papa\sFoto|pavuk|pcBrowser|Program\sShareware\s1|ProPowerbot/2.14|ProWebWalker|ProWebWalker|\
psbot/0.1|QueryN.Metasearch|ReGet|RepoMonkey|RMA|SiteSnagger|SlySearch|SmartDownload|Spankbot|spanner|\
Superbot|SuperHTTP|Surfbot|suzuran|Szukacz/1.4|tAkeOut|Teleport|Teleport\sPro|Telesoft|The.Intraformant|\
TheNomad|TightTwatbot|Titan|toCrawl/UrlDispatcher|toCrawl/UrlDispatcher|True_Robot|turingos|\
Turnitinbot/1.5|URLy.Warning|VCI|VoidEYE|WebAuto|WebBandit|WebCopier|WebEMailExtrac.*|WebEnhancer|\
WebFetch|WebGo\sIS|Web.Image.Collector|Web\sImage\sCollector|WebLeacher|WebmasterWorldForumbot|\
WebReaper|WebSauger|Website\seXtractor|Website.Quester|Website\sQuester|Webster.Pro|WebStripper|\
Web\sSucker|WebWhacker|WebZip|Wget|Widow|[Ww]eb[Bb]andit|WWW-Collector-E|WWWOFFLE|\
Xaldon\sWebSpider|Xenu's|Zeus) [NC]
RewriteRule .? - [F]

Block Bad Bots with SetEnvIfNoCase

ErrorDocument 403 /403.html
 
# IF THE UA STARTS WITH THESE
SetEnvIfNoCase ^User-Agent$ .*(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(emailwolf|erocrawler|exabot|eyenetie|filehound|flashget|flunky) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(frontpage|getright|getweb|go.?zilla|go-ahead-got-it|gotit|grabnet) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(grafula|harvest|hloader|hmview|httplib|httrack|humanlinks|ilsebot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(infonavirobot|infotekies|intelliseek|interget|iria|jennybot|jetcar) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(joc|justview|jyxobot|kenjin|keyword|larbin|leechftp|lexibot|lftp|libweb) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(likse|linkscan|linkwalker|lnspiderguy|lwp|magnet|mag-net|markwatch) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(mata.?hari|memo|microsoft.?url|midown.?tool|miixpc|mirror|missigua) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(mister.?pix|moget|mozilla.?newt|nameprotect|navroad|backdoorbot|nearsite) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(net.?vampire|netants|netcraft|netmechanic|netspider|nextgensearchbot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(attach|nicerspro|nimblecrawler|npbot|octopus|offline.?explorer) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(offline.?navigator|openfind|outfoxbot|pagegrabber|papa|pavuk) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(pcbrowser|php.?version.?tracker|pockey|propowerbot|prowebwalker) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(psbot|pump|queryn|recorder|realdownload|reaper|reget|true_robot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(repomonkey|rma|internetseer|sitesnagger|siphon|slysearch|smartdownload) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(snake|snapbot|snoopy|sogou|spacebison|spankbot|spanner|sqworm|superbot) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(superhttp|surfbot|asterias|suzuran|szukacz|takeout|teleport) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(telesoft|the.?intraformant|thenomad|tighttwatbot|titan|urldispatcher) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(turingos|turnitinbot|urly.?warning|vacuum|vci|voideye|whacker) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(widow|wisenutbot|wwwoffle|xaldon|xenu|zeus|zyborg|anonymouse) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(libwww-perl|aesop_com_spiderman) HTTP_SAFE_BADBOT
Deny from env=HTTP_SAFE_BADBOT

Original Bad Bot / Web Scraper List

1. WebBandit
2. 2icommerce
3. Accoona
4. ActiveTouristBot
5. adressendeutschland
6. aipbot
7. Alexibot
8. Alligator
9. AllSubmitter
10. almaden
11. anarchie
12. Anonymous
13. Apexoo
14. Aqua_Products
15. asterias
16. ASSORT
17. ATHENS
18. AtHome
19. Atomz
20. attache
21. autoemailspider
22. autohttp
23. b2w
24. bew
25. BackDoorBot
26. Badass
27. Baiduspider
28. Baiduspider+
29. BecomeBot
30. berts
31. Bitacle
32. Biz360
33. Black.Hole
34. BlackWidow
35. bladder fusion
36. Blog Checker
37. BlogPeople
38. Blogshares Spiders
39. Bloodhound
40. BlowFish
41. Board Bot
42. Bookmark search tool
43. BotALot
44. BotRightHere
45. Bot mailto:craftbot@yahoo.com
46. Bropwers
47. Browsezilla
48. BuiltBotTough
49. Bullseye
50. BunnySlippers
51. Cegbfeieh
52. CFNetwork
53. CheeseBot
54. CherryPicker
55. Crescent
56. charlotte/
57. ChinaClaw
58. Convera
59. Copernic
60. CopyRightCheck
61. cosmos
62. Crescent
63. c-spider
64. curl
65. Custo
66. Cyberz
67. DataCha0s
68. Daum
69. Deweb
70. Digger
71. Digimarc
72. digout4uagent
73. DIIbot
74. DISCo
75. DittoSpyder
76. DnloadMage
77. Download
78. dragonfly
79. DreamPassport
80. DSurf
81. DTS Agent
82. dumbot
83. DynaWeb
84. e-collector
85. EasyDL
86. EBrowse
87. eCatch
88. ecollector
89. edgeio
90. efp@gmx.net
91. EirGrabber
92. Email Extractor
93. EmailCollector
94. EmailSiphon
95. EmailWolf
96. EmeraldShield
97. Enterprise_Search
98. EroCrawler
99. ESurf
100. Eval
101. Everest-Vulcan
102. Exabot
103. Express
104. Extractor
105. ExtractorPro
106. EyeNetIE
107. FairAd
108. fastlwspider
109. fetch
110. FEZhead
111. FileHound
112. findlinks
113. Flaming AttackBot
114. FlashGet
115. FlickBot
116. Foobot
117. Forex
118. Franklin Locator
119. FreshDownload
120. FrontPage
121. FSurf
122. Gaisbot
123. Gamespy_Arcade
124. genieBot
125. GetBot
126. Getleft
127. GetRight
128. GetWeb!
129. Go!Zilla
130. Go-Ahead-Got-It
131. GOFORITBOT
132. GrabNet
133. Grafula
134. grub
135. Harvest
136. Hatena Antenna
137. heritrix
138. HLoader
139. HMView
140. holmes
141. HooWWWer
142. HouxouCrawler
143. HTTPGet
144. httplib
145. HTTPRetriever
146. HTTrack
147. humanlinks
148. IBM_Planetwide
149. iCCrawler
150. ichiro
151. iGetter
152. Image Stripper
153. Image Sucker
154. imagefetch
155. imds_monitor
156. IncyWincy
157. Industry Program
158. Indy
159. InetURL
160. InfoNaviRobot
161. InstallShield DigitalWizard
162. InterGET
163. IRLbot
164. Iron33
165. ISSpider
166. IUPUI Research Bot
167. Jakarta
168. java/
169. JBH Agent
170. JennyBot
171. JetCar
172. jeteye
173. jeteyebot
174. JoBo
175. JOC Web Spider
176. Kapere
177. Kenjin
178. Keyword Density
179. KRetrieve
180. ksoap
181. KWebGet
182. LapozzBot
183. larbin
184. leech
185. LeechFTP
186. LeechGet
187. leipzig.de
188. LexiBot
189. libWeb
190. libwww-FM
191. libwww-perl
192. LightningDownload
193. LinkextractorPro
194. Linkie
195. LinkScan
196. linktiger
197. LinkWalker
198. lmcrawler
199. LNSpiderguy
200. LocalcomBot
201. looksmart
202. LWP
203. Mac Finder
204. Mail Sweeper
205. mark.blonin
206. MaSagool
207. Mass
208. Mata Hari
209. MCspider
210. MetaProducts Download Express
211. Microsoft Data Access
212. Microsoft URL Control
213. MIDown
214. MIIxpc
215. Mirror
216. Missauga
217. Missouri College Browse
218. Mister
219. Monster
220. mkdb
221. moget
222. Moreoverbot
223. mothra/netscan
224. MovableType
225. Mozi!
226. Mozilla/22
227. Mozilla/3.0 (compatible)
228. Mozilla/5.0 (compatible; MSIE 5.0)
229. MSIE_6.0
230. MSIECrawler
231. MSProxy
232. MVAClient
233. MyFamilyBot
234. MyGetRight
235. nameprotect
236. NASA Search
237. Naver
238. Navroad
239. NearSite
240. NetAnts
241. netattache
242. NetCarta
243. NetMechanic
244. NetResearchServer
245. NetSpider
246. NetZIP
247. Net Vampire
248. NEWT ActiveX
249. Nextopia
250. NICErsPRO
251. ninja
252. NimbleCrawler
253. noxtrumbot
254. NPBot
255. Octopus
256. Offline
257. OK Mozilla
258. OmniExplorer
259. OpaL
260. Openbot
261. Openfind
262. OpenTextSiteCrawler
263. Oracle Ultra Search
264. OutfoxBot
265. P3P
266. PackRat
267. PageGrabber
268. PagmIEDownload
269. panscient
270. Papa Foto
271. pavuk
272. pcBrowser
273. perl
274. PerMan
275. PersonaPilot
276. PHP version
277. PlantyNet_WebRobot
278. playstarmusic
279. Plucker
280. Port Huron
281. Program Shareware
282. Progressive Download
283. ProPowerBot
284. prospector
285. ProWebWalker
286. Prozilla
287. psbot
288. psycheclone
289. puf
290. PushSite
291. PussyCat
292. PuxaRapido
293. Python-urllib
294. QuepasaCreep
295. QueryN
296. Radiation
297. RealDownload
298. RedCarpet
299. RedKernel
300. ReGet
301. relevantnoise
302. RepoMonkey
303. RMA
304. Rover
305. Rsync
306. RTG30
307. Rufus
308. SAPO
309. SBIder
310. scooter
311. ScoutAbout
312. script
313. searchpreview
314. searchterms
315. Seekbot
316. Serious
317. Shai
318. shelob
319. Shim-Crawler
320. SickleBot
321. sitecheck
322. SiteSnagger
323. Slurpy Verifier
324. SlySearch
325. SmartDownload
326. sna-
327. snagger
328. Snoopy
329. sogou
330. sootle
331. So-net” bat_bot
332. SpankBot” bat_bot
333. spanner” bat_bot
334. SpeedDownload
335. Spegla
336. Sphere
337. Sphider
338. SpiderBot
339. sproose
340. SQ Webscanner
341. Sqworm
342. Stamina
343. Stanford
344. studybot
345. SuperBot
346. SuperHTTP
347. Surfbot
348. SurfWalker
349. suzuran
350. Szukacz
351. tAkeOut
352. TALWinHttpClient
353. tarspider
354. Teleport
355. Telesoft
356. Templeton
357. TestBED
358. The Intraformant
359. TheNomad
360. TightTwatBot
361. Titan
362. toCrawl/UrlDispatcher
363. True_Robot
364. turingos
365. TurnitinBot
366. Twisted PageGetter
367. UCmore
368. UdmSearch
369. UMBC
370. UniversalFeedParser
371. URL Control
372. URLGetFile
373. URLy Warning
374. URL_Spider_Pro
375. UtilMind
376. vayala
377. vobsub
378. VCI
379. VoidEYE
380. VoilaBot
381. voyager
382. w3mir
383. Web Image Collector
384. Web Sucker
385. Web2WAP
386. WebaltBot
387. WebAuto
388. WebBandit
389. WebCapture
390. webcollage
391. WebCopier
392. WebCopy
393. WebEMailExtrac
394. WebEnhancer
395. WebFetch
396. WebFilter
397. WebFountain
398. WebGo
399. WebLeacher
400. WebMiner
401. WebMirror
402. WebReaper
403. WebSauger
404. WebSnake
405. Website
406. WebStripper
407. WebVac
408. webwalk
409. WebWhacker
410. WebZIP
411. Wells Search
412. WEP Search 00
413. WeRelateBot
414. Wget
415. WhosTalking
416. Widow
417. Wildsoft Surfer
418. WinHttpRequest
419. WinHTTrack
420. WUMPUS
421. WWWOFFLE
422. wwwster
423. WWW-Collector
424. Xaldon
425. Xenu's
426. Xenus
427. XGET
428. Y!TunnelPro
429. YahooYSMcm
430. YaDirectBot
431. Yeti
432. Zade
433. ZBot
434. zerxbot
435. Zeus
436. ZyBorg

Blocking Bad Bots and Scrapers with .htaccess originally appeared on AskApache.com

Alot of people have asked me about the best way to block specific IP addresses that are attacking their servers. No easy answer for sure. Here's what I came up with.

Per-domain Error.log and Access.log checks

This shell script analyzes a specific domains apache logs, both error and access, and then finds the IP address that hit your server the most. Then it checks for a a reverse DNS for each of the IP addresses, if it doesn't have a reverse DNS, AND it results in a positive to one of the 3 other tests its added to a bad IP list.

Generates .htaccess blocking code

Once the tests have concluded (on my machine it takes maybe 5 minutes) you have the choice to view an automatically generated .htaccess file that you can copy into your real .htaccess file, full of the correct code to Block the bad IP's. You can tell them bye bye with a 403 Forbidden.

IP Abuse / .htaccess Blocking Script

#!/bin/sh
 
# User-contributed script. Not sponsored by DreamHost.
# Script created 2008-01-16 by AskApache (www.askapache.com)
 
### SHELL OPTIONS
set +o noclobber  # allowed to clobber files
set +o noglob     # globbing on
set +o xtrace     # change to - to enable tracing
set +o verbose    # change to - to enable verbose debugging
set -e            # abort on first error
 
# directory where log files, reports, and generated .htaccess files will be saved
TMPDI="$HOME/ip_abuse"
 
function exitt(){
 case $TERM in
  xterm*|vt*|ansi|rxvt|gnome*) echo -e "\033]0;$USER@`hostname`: $HOME\007" ;;
 esac
}
 
function ok_continue()
{ echo -en "\n\033[?25l\033[30;42m[ Press any key to continue ]\033[0m\n" ;read -n 1 ans;echo -en "\033[?25h"; }
 
function test_title()
{ echo -en "\n\n\033[0;32m\n>>>"; echo -e "\033[1;37m $1 \033[0m \n"; echo -e "\n\n[ ${1} ]\n" >> $REPORT; }
 
function error_abuse(){
 clear; title
 case "$2" in
  connlimit|1*) test_title "CONCURRENT CONNECTION TEST"; future=2
   echo -e "Shows IP's making more than 20 requests concurrently.\n\n"
   cat $TMPDI/$YD/logs/e* |grep 'concurrent\ connection\ limit'|awk -F ']' '{print $3}' |
   awk '{print $2}'|sort|uniq -c |sort -nr|sed 's/^ *//'|egrep "[0-9]{2}+\ " > $TMPDI/$YD/logs/out.txt ;;
  access|2*) test_title "TOP 50 IP TEST"; future=3
   echo -e "Displays the top 50 unique IP addresses that access your site."
   echo -e "If they don't have a reverse DNS maybe they should be blocked.\n\n"
   cat $TMPDI/$YD/logs/a* |awk '{print $1|"sort|uniq -dc|sort -nr"}' |egrep "[0-9]{3}+\ "|
   awk '{print $1,$2}' > $TMPDI/$YD/logs/out.txt ;;
  internal|3*) test_title "INTERNAL RECURSION TEST"; future="report"
   echo -e "Shows the IP's that triggered an Internal Recursion Error,"
   echo -e "meaning that their is a looping problem on your server.\n"
   cat $TMPDI/$YD/logs/e* |grep 'LimitInternalRecursion'|awk -F ']' '{print $3}'|awk '{print $2}' |
  sort|uniq -c|sort -nr|sed 's/^ *//'|egrep "[0-9]{2}+\ " > $TMPDI/$YD/logs/out.txt ;;
 esac
 
 echo -e "\033[?25l";t=0; h=0; cat $TMPDI/$YD/logs/out.txt |
 while read a
 do
 if [ $t -lt 50 ];then
  n=`echo "$a"|awk '{print $1}'`; ip=`echo "$a"|awk '{print $2}'`
  host=`nice -n 19 host -qQ -s 1 "${ip}" 2>&1|tr '\n' '\t'|awk '{print $2}'`;
  case "$host" in
   does) echo -en "\033[0;33m"; echo -e "$ip" >> $REPORT
    host=${host/does/!!!POSSIBLE-ABUSE!!!}; (( t++ )); (( h++ )); ips=" $ip$ips" ;;
   not) echo -en "\033[0;33m"; echo -e "$ip" >> $REPORT
    host=${host/not/!!!POSSIBLE-ABUSE!!!}; (( t++ )); (( h++ )); ips=" $ip$ips" ;;
   PTR) echo -en "\033[0;33m"; echo -e "$ip" >> $REPORT
    host=${host/PTR/!!!POSSIBLE-ABUSE!!!}; (( t++ )); (( h++ )); ips=" $ip$ips" ;;
   .) echo -en "\033[0;33m"; echo -e "$ip" >> $REPORT
    host=${host/\./!!!POSSIBLE-ABUSE!!!}; (( t++ )); (( h++ )); ips=" $ip$ips" ;;
  esac
  echo -en " ${n}\t${ip}\t${host}\033[0m\n"
  if [ $h -gt 3 ]; then
  echo -e "Deny from$ips" >> $TMPDI/$YD/.htaccess; h=0; ips=" ";
  fi
fi
done
[ $h -gt 0 ] && echo -e "Deny from$ips" >> $TMPDI/$YD/.htaccess
ok_continue; exec sh $0 "$1" "$future"
}
 
function menu(){
 PS3="`echo -e '\033[0;36m'`Please Select a Domain To Test: `echo -e '\033[0m'`"; echo -ne "\033[0m"
 select v
 do
  YD="$v"
  mkdir -p -m 0755 $TMPDI/$YD/reports
  mkdir -p -m 0755 $TMPDI/$YD/logs
  echo "" > $TMPDI/$YD/logs/out.txt
  echo "" > $TMPDI/$YD/.htaccess
  echo -e "## IP-ABUSE-LOOKUP\nOrder Allow,Deny\nAllow from All" > $TMPDI/$YD/.htaccess
  ELOG="$TMPDI/$YD/logs/error.log"
  ALOG="$TMPDI/$YD/logs/access.log"
  REPORT=$(echo -en "$TMPDI/$YD/reports/`date +%mx%dx%y`.txt");
  echo "" > $REPORT; echo -e "GENERATED REPORT FOR $YD \n`date`\n" >> $REPORT
  clear; title; cd ~/logs/$YD/http
  if [ ! -f $ELOG ]; then
   test_title "Creating Error log"
   `nice -n 19 gunzip -dc e*.gz | split -b 1m -a 4 -d - $TMPDI/$YD/logs/e.`; wait
   cp error.log $TMPDI/$YD/logs; echo -e "\033[0;31m [ DONE ]\n\n"
  fi
  if [ ! -f $ALOG ]; then
   test_title "Creating access log"
   `nice -n 19 gunzip -dc a*.gz | split -b 5m -a 4 -d - $TMPDI/$YD/logs/a.`; wait
   cp access.log $TMPDI/$YD/logs; echo -e "\033[0;31m [ DONE ]\n\n"
  fi
  cd $HOME; exec sh $0 "$YD" "1"
 done
}
 
function show_report(){
 clear; title;
 PS3="`echo -e '\033[0;36m'`Please select a course of action: `echo -e '\033[0m'`"; echo -ne "\033[0m"
 select v in "View Report" "View .htaccess" "Quit"
 do
  case "$v" in
   *Report) clear; title; test_title "VIEWING $REPORT"; cat $REPORT ;;
   *htaccess) clear; title; test_title "VIEWING $TMPDI/$YD/.htaccess"; cat $TMPDI/$YD/.htaccess ;;
   Quit) break;;
  esac
 done
exit 0
}
 
function title(){
 # pretty sweet!
 echo -e "\033[1;30m __________________________________________________________________________ "
 echo -e "|\033[1;32m                 ___                     __ __         __                 \033[1;30m|"
 echo -e "|\033[1;32m                / _ \_______ ___ ___ _  / // /__  ___ / /_                \033[1;30m|"
 echo -e '|\033[1;32m               / // / __/ -_) _ `/  ` \/ _  / _ \(_-</ __/                \033[1;30m|'
 echo -e "|\033[0;32m              /____/_/  \__/\_,_/_/_/_/_//_/\___/___/\__/                 \033[1;30m|"
 echo -e "|                                                                          |"
 echo -e "|            \033[1;37mDREAMHOST IP ABUSE DETECTION SCRIPT VERSION 0.1\033[1;30m               |";
 echo -e "\033[1;30m __________________________________________________________________________ \033[0m\n\n"
}
 
# catch non-kill exit to reset term / ncurses
trap exitt EXIT
 
# set window title if client is capable
case $TERM in
 xterm*|vt*|ansi|rxvt|gnome*) echo -e "\033]0;DREAMHOST IP ABUSE DETECTION SCRIPT\007" ;;
esac
 
if [ $# -lt 1 ]; then
 clear; title; [ -d $TMPDI ] || mkdir -m 755 $TMPDI
 cd ~/logs; DOMAINS=( `ls ~/logs/` ); cd $OLDPWD; menu ${DOMAINS[@]}
else
 YD="${1}"
 mkdir -p -m 0755 $TMPDI/$YD/reports; mkdir -p -m 0755 $TMPDI/$YD/logs
 touch $TMPDI/$YD/logs/out.txt;
 ELOG="$TMPDI/$YD/logs/error.log"
 ALOG="$TMPDI/$YD/logs/access.log"
 REPORT=$(echo -en "$TMPDI/$YD/reports/`date +%mx%dx%y`.txt");
 
 if [ "$2" == "report" ]; then
  show_report "$1" "$2"
 else
  error_abuse "$1" "$2"
 fi
fi
 
exit 0

Read more on the DreamHost wiki page: Block IP Abuse.

IP Abuse Detection for DreamHost originally appeared on AskApache.com