Ok, sup. I really felt I had to get this out of the way, because I have a whole stack of drafts waiting to be published, but I realized that not many people will benefit from all the advanced optimizations and tricks I'm writing unless they get a basic understanding of some of the tools I'm using. I decided to write a series of articles explaining how I optimize servers for speed because lately I've been getting a lot more people wanting to hire me to do that. I take on projects when I can but there is clearly a need out here on the net for some self-help. The momentum is swinging more and more towards VPS type of web hosting, and I would say that 99% of those customers are getting supremely ripped off, which goes against the foundation of the web.

Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I'm not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn't it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won't be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU's handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore "processes") are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place... because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I've opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that's completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 - tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching - CPU processing
2. DISK IO Bottleneck - Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 - Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it's important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N - add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not "punchy" enough), the scheduler was decoupled from 'time slice' and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization "split" between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be "multiplicative" (or exponential) - that way it does not matter which nice level you start out from, the 'relative result' will always be the same.

The third complaint (of negative nice levels not being "punchy" enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID's 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following "real-time" policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following "normal" (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload's tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn't set a specific io priority. This is the default scheduling class for any process that hasn't asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it's directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn't really apply here.

ionice

ionice - get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c - The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n - The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p - Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs 'bash' as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]
 
-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities - schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces -- thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I've found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn't be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit - get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell's resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft' resource limit

-H

use the `hard' resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process's data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft', `hard', and `unlimited' stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process's data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process's nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 - rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process's resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler's nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on AskApache.com

One of the ways I speed up AskApache.com is by downloading all the external javascript files to my server and then serve them from my own server instead of the external site. Currently I'm using this method to serve the Google Analytics javascript file, and the Quantcast javascript file.

Speed Benefit

The speed benefit occurs because normally a site visitor has to perform a DNS query for both the google-analytics.com and quantcast.com servers, create a connection to each server, and then download the files. By hosting these scripts on your own server you remove all the extra DNS queries, and by hosting all the files on your single server you enable advanced HTTP 1/1 protocol features like Pipelining the requests, which means a single connection for multiple files.

Cache and Compression Control

Serving these javascript files from your own server gives you complete control over the caching and compression settings of those files, which may not seem like a big deal... unless you know about the advanced caching and compression techniques available to make your site super-fast.

Caching Control

By using the mod_rewrite cache hack like I do, you can make sure that a fresh version of the .js file is used by your visitors.

Compression Control

You can also specify advanced compression for these files, which could mean using Apache's mod_deflate or mod_gzip modules, or you can even use a compression tool like Dojo's Shrinksafe, Packer, JSMin, or YUICompressor to further compress the javascript files.

Automated Downloading with Crontab

Using crontab I tell my web server to run this shell script every day, to make sure I have updated versions of all the files. Here's the crontab entry I use to run this shell script every day.

@daily /bin/sh /home/cron-scripts/grab_javascripts.sh 2>&1 &>/dev/null

Shell Script to Download JS

I created a simple Bash shell-script to download the javascript files. All you need to do is modify the variables to save to your directory and send your site as the Referring URL and you are good to go. You can add as many files as you want by adding them to the JSFILE array.

You can also download the shell-script: grab_javascripts.sh

#!/bin/bash
# 2008-09-30
umask 022
VER="2.0"
 
### Set as the referring url for downloads
SITE=http://www.askapache.com/
 
### Directory to save the downloaded files
JSD=$HOME/j/
 
### The files to download
JSFILE[0]=http://www.google-analytics.com/ga.js
 
### run script with args to turn on debugging
[[ $# -ne 0 ]] && set -o xtrace && set -o verbose
 
### SHELL OPTIONS
set +o noclobber # allowed to clobber files
set +o noglob # globbing on
set -e # abort on first error
shopt -s extglob
 
#--=--=--=--=--=--=--=--=--=--=--#
# pt
#==-==-==-==-==-==-==-==-==-==-==#
function pt(){
  case ${1:-d} in
   i) echo -e "${C6}=> ${C4}${2} ${C0}"; ;;
   m) echo -en "\n\n${C2}>>> ${C4}${2} ${C0}\n\n"; ;;
   *) echo -e "\n${C8} DONE ${C0} \n"; ;;
  esac
}
 
#=# CATCH SCRIPT KILLED BY USER
trap 'kill -9 $$' SIGHUP SIGINT SIGTERM
 
#=# MAKE MAIN SCRIPT NICE AS POSSIBLE SINCE IT DOESNT DO MUCH
renice 19 -p $$ &>/dev/null
 
#=# TURNS ON COLORING ONLY FOR TERMS THAT CAN SUPPORT IT
C="\033[";C0=;C1=;C2=;C3=;C4=;C5=;C6=;C7=;C8=;C9=;
  C0="${C}0m";C1="${C}1;30m";C2="${C}1;32m";C3="${C}0;32m";C4="${C}1;37m"
  C5="${C}0;36m";C6="${C}1;35m";C7="${C}0;37m";C8="${C}30;42m";C9="${C}1;36m";
clear
echo -e "${C1} __________________________________________________________________________ "
echo -e "| ${C2}             ___       __    ___                 __             ${C1}         |"
echo -e "| ${C2}            / _ | ___ / /__ / _ | ___  ___ _____/ /  ___        ${C1}         |"
echo -e "| ${C2}           / __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)       ${C1}         |"
echo -e "| ${C3}          /_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/        ${C1}         |"
echo -e "| ${C3}                               /_/                              ${C1}         |"
echo -e "|                                                                          |"
echo -e "|                 ${C4} LOCAL JAVASCRIPT FILES SCRIPT Version ${VER} ${C1}              |"
echo -e "${C1} __________________________________________________________________________ ${C0} \n\n"
 
#=# BUILD INCOMING DIRS
[[ ! -d "${JSD}" ]] && pt m "BUILDING DIRS" && mkdir -p $JSD &>/dev/null && pt i "CREATED $JSD" && pt
 
#=# DOWNLOAD JAVASCRIPT FILES
cd $JSD && pt m "DOWNLOADING JAVASCRIPT FILES"
for theurl in "${JSFILE[@]}"; do
pt i "${theurl}"
curl -m 60 --connect-timeout 10 --retry 10 --retry-delay 180 -s -S -L -e '${SITE}' -A 'Mozilla/5.0' -O "${theurl}"
done
pt && cd $OLDPWD
 
######## curl options
# -S Show error. With -s, make curl show errors when they occur
# -s Silent mode. Don't output anything
# -e  Referer URL (H)
# -H <line> Custom header to pass to server (H)
# -L  Follow Location: hints (H)
# -A <string> User-Agent to send to server (H)
# -m <seconds> Maximum time allowed for the transfer
# --connect-timeout <seconds> Maximum time allowed for connection
# --globoff  Disable URL sequences and ranges using {} and []
# -O  Write output to a file named as the remote file
# --retry <num>  Retry request <num> times if transient problems occur
# --retry-delay <seconds> When retrying, wait this many seconds between each
# --retry-max-time <seconds> Retry only within this period
#############################################################################
 
exit 0

Serve External Javascript Files locally for Increased Speed originally appeared on AskApache.com

Ok peeps, one of the coolest hacks yet. If you have multiple PC's/Monitors at your workstation like I do, it can be helpful to display various information on one screen while you work on another.

This article shows how I continuously scroll the logs for a server/site I am working on, thus saving me a lot of time by providing real-time debugging on a separate screen. Not only does this scroll the latest log entries as they are created, it displays them in color using syntax highlighting to make your logs incredibly easy to understand and parse.

Example Output

The thing to realize is that this output is continuously scrolling on your monitor, and using some cool linux shell tricks you can make it output at a certain speed and show a certain number of lines. Another cool feature is you can display multiple files at the same time, and the filename will be output for each file above the log output.

What Logs

I've explained in various articles on this site how to create and use various custom log files using .htaccess files and other tricks for non-root users. I use shared hosting all the time too you know... Here are some log files you can generally use if you can use .htaccess files (if you can control .htaccess you can control a binary php-cgi, which could be a shell script thats execs the php binary of your choice with your own environment variables, which is a workaround to getting a custom php.ini, which is how you specify a php error log).

• php error log
• mod_security audit log
• mod_security debug log
• apache error log
• apache access log

Any log file can be used with this method, actually ANY file containing text can be used with this method, even fifo pipes as we will see in a bit ;)

Installation

PCRE

First you need PCRE - Perl Compatible Regular Expressions, which you can download here. Note that I had to install pcre version 6.7 to get it to work with ccze.

CCZE

Now you need CCZE, which colorizes and outputs (emulating tail) the log files.

CCZE is a robust and modular log colorizer with plugins for apm, exim, fetchmail, httpd, postfix, procmail, squid, syslog, ulogd, vsftpd, xferlog, and more.

Post-Install Setup

Ok so once you've installed ccze you can start using it right away, or you can continue reading to see how I've set it up for readable scrolling. If the following setup isn't to your taste, you can always just use wtail, a colorless multi-file scroller that uses separate scrolling windows in your terminal similar to the screen program.

Create Logs Folder

First I created a folder called /logs/ for my site, then for all subsequent commands chdir to it.

$ mkdir /home/site.com/logs
$ cd /home/site.com/logs

Create Symlinks to Logs

In the new logs folder I created soft symlinks to all the various log files, not neccessary but makes everything much easier and organized.

$ ln -s /actual/logs/site.com/http/access.log access.log
$ ln -s /actual/logs/site.com/http/error.log error.log
$ ln -s /home/site.com/php_error.log php_error.log
$ ln -s /home/site.com/modsec_audit.log modsec_audit.log
$ ln -s /home/site.com/modsec_debug.log modsec_debug.log

Make a fifo Pipe

FIFO stands for first-in-first-out and is a somewhat complex feature of linux/bsd/unix shells that let you send data into it and read the data that comes out of it, just like a pipe.

$ mkfifo pipe

Scrolling the Logs

First cd to the log directory you created.

$ cd /home/site.com/logs

Now we will use the tail program to output 120 lines of each of these log files every half of a second, only displaying changes/new log entries. We send that output into the fifo pipe we created using shell redirection and then instruct the command to run in the background using the ampersand to access built-in (bash/sh) job control.

$ tail -s .5 -n 120 -f access.log php_error.log error.log modsec_audit.log >pipe &

Displaying the Colorized Scrolling Output

So every .5 seconds the tail command outputs any new log entries from those files into the fifo pipe, so now we need to use the ccze program to use the fifo pipe as its input log file. Normally you run ccze like the cat command and it just outputs the input colorized, by using tail and a fifo pipe we are able to make this awesome trick work.

To current TTY

$ ccze <pipe

To any TTY

$ ccze < pipe >/dev/pts/2 &

Save All Colored Output

This appends the colorized output to the file in the CWD, super.log

$ ccze -A <pipe | tee -a super.log

Kill the Scrolling

Immediately kills any processes used by tail and ccze.

$ pkill -9 ccze\|tail

Going Further. Hackers only.

Here are some other examples of using ccze with fifo pipes, more as an excercise than anything practical as most lines don't work, I just grabbed them from my shell history file.

tail -f access.log >pipe & ccze <pipe | tee -a super.log >/dev/pts/2
tail -f access.log >pipe & ccze <pipe | tee -a super.log >/dev/pts/2 &
 
exec 3<> pipe; while ccze >/dev/pts/2 <&3; do tail -f access.log >pipe; done; exec 3>&-
exec 3<> pipe; ccze <&3 >/dev/pts/2 & ; tail -f access.log >pipe; exec 3>&-
exec 3<> pipe; ccze <&3 >/dev/pts/2 & ; tail -f access.log >pipe
exec 3<> pipe; ccze <&3 >/dev/pts/2 & tail -f access.log >pipe
ccze <&3 >/dev/pts/2 &
ccze 0<&3 |tee -a super.log > /dev/pts/2 &
ccze <&3 |tee -a super.log > /dev/pts/2 &
ccze <pipe |tee -a /dev/pts/2 & tail -s .5 -n 120 -f php_error.log error.log modsec_audit.log >pipe &
ccze | tee -a /dev/pts/2 <pipe
 
ccze < tee -a /dev/pts/2 pipe >/dev/pts/2
ccze <|tee -a /dev/pts/2 pipe >/dev/pts/2
ccze <tee -a /dev/pts/2 <pipe
ccze <pipe >> tee -a super.log >/dev/pts/2
cat <pipe | ccze
ccze <pipe >>/dev/pts/2 &
 
tail -fq access.log | ccze  &>/dev/pts/1
tail -qf access.log |ccze &>/dev/pts/1 &
tail -f access.log | ccze > $SSH_TTY
 
( ccze > /dev/pts/2 )<pipe
( ccze > /dev/pts/2 )<pipe
( ccze > /dev/pts/2 )<pipe & tail -f access.log | pipe
( ccze > /dev/pts/2 )<pipe & tail -f access.log | ./pipe
 
ccze <pipe > /dev/pts/2 & tail -f access.log >pipe
ccze  <pipe> /dev/pts/2 & tail -f access.log >pipe
ccze  < pipe > /dev/pts/2 & tail -f access.log >pipe
ccze < pipe >/dev/pts/2 & tail -f access.log >pipe
ccze < pipe >/dev/pts/2 & tail -f access.log >pipe &
ccze < pipe >/dev/pts/2 & tail -n 40 -s 2 -f error.log modsec_audit.log php_error.log  >pipe &
ccze < pipe >| tee -a super.log | >/dev/pts/2 & tail -n 80 -s 1 -f error.log modsec_audit.log php_error.log  >pipe &
ccze < pipe >/dev/pts/2 & tail -n 100 -s 1 -f error.log modsec_audit.log php_error.log >pipe &
ccze < pipe >/dev/pts/2 & tail -n 100 -s 1 -f error.log modsec_audit.log php_error.log >pipe &
 
tail -f access.log | ccze > pipe & /dev/pts/2/<pipe
tail -f access.log | ccze > pipe & /dev/pts/2<pipe
 
ccze < $(tail -n 20 -s 1 -f access.log)
ccze < | $(tail -n 20 -s 1 -f access.log)
ccze < |$(tail -n 20 -s 1 -f access.log)

Shell Script

I also hacked together a little shell script that you may wish to hack to your needs. Its actually pretty sweet if you can figure it out.

#!/bin/bash -l
set -o xtrace
renice -p $$ 15
 
pkill -9 tee\|ccze &>/dev/null || echo -n
disown -a &>/dev/null || echo
 
[[ -r "~/pipe" ]] && rm -rf ~/pipe
 
mkfifo ~/pipe
 
cd /home/askapache.com/logs
 
[[ -r "superlog.log" ]] && echo "Found old logfile, moving." && mv superlog.log `command ls|wc -l`-superlog.log
 
ccze <$HOME/pipe > $SSH_TTY & tail -n 150 -s .5 -f  php_error.log error.log modsec_audit.log >$HOME/pipe &
disown %2; && disown %3;
 
disown -a
 
sleep 60 &
disown $! || disown %1
 
for i in `seq 1 4`;do echo -n $'\a'; sleep 1;done
kill -9 $J1
kill -9 $J2
 
logout
exit 0

Elite Log File Scrolling with Color Syntax originally appeared on AskApache.com

Because backups contain all your sensitive information, its smart to encrypt any sql backups.. and while we're at it, also encrypt any site backups.

This simple shell-script is a useful and easy way to securely backup your wordpress and/or phpBB site files and database without confusing you. Just generate a GPG key once, enter in 3 settings once, and from then on it runs without any user-input whenever you want.

What it Does

When run, this script asks you for the location of your websites document root and the location of your wp-config.php or config.php file. It also asks you for your encryption UID. Then this script saves those settings in a file called .sbackup so that the next time you run the script it will run without having to re-enter that information, making it nice for cronjobs or quick and easy on-demand backups. Another cool feature that I added is this script automatically parses your wp-config.php file for the mysql database name, user, host, and password, meaning you don't have to compromise your security or take the time to type those settings in manually.

What is Backed Up

This script creates a tarred and gzipped archive of your entire document root in the folder ~/backups/domain.com/domain.com-date.tgz and also creates a backup of your WordPress database and phpBB database in a format that is ideal for restoring from. Both of these files are then encrypted using your GPG key and can then be safely downloaded as a password and key is required to decrypt them.

Generating a GPG Key

If you don't already have one setup for your shell account run this command remembering the uid which you will enter in the shell script.

gpg --gen-key

Decrypting Files

gpg -r UID --output FILENAME.tgz --decrypt FILENAME.tgz.asc

The Shell Script

site-backup.sh

#!/bin/bash
# SiteBack Version 3.3, 2008-12-17
# GNU Free Documentation License 1.2
# 12-17-08 - AskApache (www.askapache.com)
umask 022
 
### SHELL OPTIONS
set +o noclobber # allowed to clobber files
set +o noglob # globbing on
set +o xtrace # change to - to enable tracing
set +o verbose # change to - to enable verbose debugging
set -e # abort on first error
shopt -s extglob
 
###########################################################################--=--=--=--=--=--=--=--=--=--=--#
###
### SETTINGS
###
###########################################################################==-==-==-==-==-==-==-==-==-==-==#
 
DT=$(date +%x); DT=${DT//\/}
DTX=$(date +%x-%H%M); DTX=${DTX//\/}
BDIR=${HOME}/backups
RUN_FILE=${BDIR}/$$.bk.log
MY_CONFIG=".sbackup"
DOMAIN=;DB_NAME=;DB_USER=;DB_PASSWORD=;DB_HOST=;APP_CONFIG=;SQL_DEST=;ARC_DEST=;ENCRYPT_USER=
E_SUCCESS=0;E_YN=0;E_YES=251;E_NO=250;E_RETURN=65;C0=;C1=;C2=;C3=;C4=;C5=;C5=;C7=
 
###########################################################################--=--=--=--=--=--=--=--=--=--=--#
###
### FUNCTIONS
###
###########################################################################==-==-==-==-==-==-==-==-==-==-==#
 
#--=--=--=--=--=--=--=--=--=--=--#
# script_title
#==-==-==-==-==-==-==-==-==-==-==#
function script_title(){
 local e="\033["
 local l=' ___________________________________________________________________ '
 
 # SET WINDOW TITLE AND COLORS IF CLIENT CAPABLE
 case $TERM in xterm*|vt*|ansi|rxvt|gnome*)
 C0="${e}0m";C1="${e}1;30m";C2="${e}1;32m";C3="${e}0;32m";C4="${e}1;37m";C5="${e}1;35m";C6="${e}30;42m"
 esac
 
 echo -e "\n${C0}$l${C1}"
 echo -e "|             ${C2}___       __    ___                 __${C1}                |"
 echo -e "|            ${C2}/ _ | ___ / /__ / _ | ___  ___ _____/ /  ___${C1}           |"
 echo -e "|           ${C2}/ __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)${C1}          |"
 echo -e "|          ${C3}/_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/${C1}           |"
 echo -e "|                               ${C3}/_/${C1}                                 |"
 echo -e "|                                                                   |"
 echo -e "|       ${C1}+--${C0} SITE BACKUP SCRIPT Version 3.3${C1}                          |"
 echo -e "${C0}$l\n\n"
}
 
#--=--=--=--=--=--=--=--=--=--=--#
# pm
#==-==-==-==-==-==-==-==-==-==-==#
function pm(){
 START=$(date +%s) && touch ${RUN_FILE}
 case "${2:-title}" in
  "title") echo -en "\n\n${C2}>>> ${C4}${1} ${C0} \n\n"; ;;
   "info") echo -e "${C5}=> ${C4}${1} ${C0}"; ;;
   "item") echo -e "${C4}-- ${C0}${1} "; ;;
 esac
}
 
#--=--=--=--=--=--=--=--=--=--=--#
# yes_no
#==-==-==-==-==-==-==-==-==-==-==#
function yes_no(){
 local ans
 echo -en "${1} [y/n] " ; read -n 1 ans
 case "$ans" in
  n|N) E_YN=$E_NO ;;
  y|Y) E_YN=$E_YES ;;
 esac
}
 
#--=--=--=--=--=--=--=--=--=--=--#
# do_sleep
#==-==-==-==-==-==-==-==-==-==-==#
function do_sleep (){
 local END DIFF
 echo -en "${C5}${3:-.}"; while [ -r "$RUN_FILE" ]; do sleep ${2:-3}; echo -en "${3:-.}"; done;
 echo -e "${C0}"; sleep 1; END=$(date +%s);DIFF=$(( $END - $START ))
 echo -e "\n${C6} [T: ${SECONDS}] COMPLETED IN ${DIFF} SEC ${C0} \n\n"; sleep 1;
 return 0;
}
 
#--=--=--=--=--=--=--=--=--=--=--#
# get_settings
#==-==-==-==-==-==-==-==-==-==-==#
function get_settings(){
 local cha HOSTED_SITES G GG
 clear; script_title
 
 if [[ -r "$MY_CONFIG" ]]; then
 
  OIFS=$IFS; while IFS=: read DOMAIN DOMAINROOT APP_CONFIG ENCRYPT_USER; do
   DOMAIN=${DOMAIN};
   DOMAINROOT=${DOMAINROOT};
   APP_CONFIG=${APP_CONFIG};
   ENCRYPT_USER=${ENCRYPT_USER};
   #E_YN=$E_YES;
   break
  done <${MY_CONFIG};
  IFS=$OIFS
 
 else
 
  gpg --list-keys|grep uid.*|awk '{print $2}'
  echo -en "\n What userid to use for encryption?  ";
  read -e ENCRYPT_USER; echo
 
  echo -en "\n What domain would you like to backup?  "; read -e DOMAIN; echo
 
  echo $PWD
  until [ -d "$DOMAINROOT" ]; do echo -en "\n Folder where config file is located?  ";
  read -e DOMAINROOT; echo; done
 
  [[ -r "$DOMAINROOT/config.php" ]] && APP_CONFIG=$DOMAINROOT/config.php && DOT=PHP
  [[ -r "$DOMAINROOT/wp-config.php" ]] && APP_CONFIG=$DOMAINROOT/wp-config.php && DOT=WP
 
  echo $PWD
  until [[ -r "$APP_CONFIG" ]]; do echo -en "\n Where is the applications config file?  "; read -e APP_CONFIG; echo; done
 
 fi
 
  [[ -r "$DOMAINROOT/config.php" ]] && APP_CONFIG=$DOMAINROOT/config.php && DOT=PHP
  [[ -r "$DOMAINROOT/wp-config.php" ]] && APP_CONFIG=$DOMAINROOT/wp-config.php && DOT=WP
 
  ### For phpBB
  if [[ "${DOT}" == "PHP" ]]; then
    GG=$(sed -e '/$db\(n\|u\|pa\|h\)/!d' -e "s/$db_\(name\|user\|passwd\|host\)\ =\ '\([^']*\).*\$/\1='\2';/g" -e 's/$db/DB_/g' ${APP_CONFIG});
    G=$(echo ${GG}|sed -e 's/DB_name/DB_NAME/g' -e 's/DB_user/DB_USER/g' -e 's/DB_passwd/DB_PASSWORD/g' -e 's/DB_host/DB_HOST/g');
  else
    G=$(sed -e "/define('DB_\(NAME\|USER\|PASSWORD\|HOST\)/!d" -e "s/[^']*'DB_\(NAME\|USER\|PASSWORD\|HOST\)'[^']*'\([^']*\)'.*$/DB_\1='\2';/g" ${APP_CONFIG})
  fi
  eval $G;
 
 mkdir -p ${BDIR}/${DOMAIN}
 SQL_DEST=${BDIR}/${DOMAIN}/${DOMAIN}-${DT}.sql;
 [[ -r "${SQL_DEST}.asc" ]] && SQL_DEST=${BDIR}/${DOMAIN}/${DOMAIN}-${DTX}.sql
 
 ARC_DEST=${BDIR}/${DOMAIN}/${DOMAIN}-${DT}.tgz;
 [[ -r "${ARC_DEST}.asc" ]] && ARC_DEST=${BDIR}/${DOMAIN}/${DOMAIN}-${DTX}.tgz
 
 if [[ "$E_YN" != "$E_YES" ]]; then
  for a in "DOMAIN" "DOMAINROOT" "APP_CONFIG" "ENCRYPT_USER" "DB_NAME" "DB_USER" "DB_PASSWORD" "DB_HOST"; do echo -e "${a}: ${!a}"; done
  echo; yes_no "ARE THESE SETTINGS CORRECT"
 fi
 
 while [[ "$E_YN" != "$E_YES" ]]; do
  for a in "DOMAIN" "DOMAINROOT" "APP_CONFIG" "ENCRYPT_USER" "DB_NAME" "DB_USER" "DB_PASSWORD" "DB_HOST"; do
   echo -en "\n (Enter for Default: ${!a} )\n ${a}:> "
   read -e cha; echo; [[ ${#cha} -gt 2 ]] && eval "$a"=$cha
  done
  yes_no "ARE THESE SETTINGS CORRECT"
 done
 
 echo -e "${DOMAIN}:${DOMAINROOT}:${APP_CONFIG}:${ENCRYPT_USER}" > $MY_CONFIG
}
 
#--=--=--=--=--=--=--=--=--=--=--#
# exit_cleanup
#==-==-==-==-==-==-==-==-==-==-==#
function exit_cleanup(){
 cd $OLDPWD
 [[ -r "${SQL_DEST}" ]] && rm ${SQL_DEST}
 [[ -r "${ARC_DEST}" ]] && rm ${ARC_DEST}
}
 
############################################################################################################
###
### MAIN CODE
###
############################################################################################################
 
#=# CATCH SCRIPT KILLED BY USER
trap exit_cleanup SIGHUP SIGINT SIGTERM
 
#=# MAKE MAIN SCRIPT NICE
renice 19 -p $$ &>/dev/null
 
cd `dirname $0`
 
get_settings
 
pm "CREATING SQL BACKUP"
mysqldump --opt -u${DB_USER} -p${DB_PASSWORD} -h ${DB_HOST} -r ${SQL_DEST} --add-drop-table ${DB_NAME} 1>&2 &>/dev/null && sleep 2 1>&2 &>/dev/null && rm ${RUN_FILE} 2>&1&
do_sleep 1 1 ":"
 
pm "ENCRYPTING SQL BACKUP"
gpg --armor --recipient ${ENCRYPT_USER} --output ${SQL_DEST}.asc --encrypt ${SQL_DEST} 1>&2 &>/dev/null && sleep 2 1>&2 &>/dev/null && rm ${RUN_FILE} 2>&1&
do_sleep 1 1 ":"; rm ${SQL_DEST}
 
pm "CREATING ARCHIVE BACKUP"
tar -czf ${ARC_DEST} . 1>&2 &>/dev/null && rm ${RUN_FILE} 2>&1&
do_sleep 1 5 ":"
 
pm "ENCRYPTING ARCHIVE BACKUP"
gpg --armor --recipient ${ENCRYPT_USER} --output ${ARC_DEST}.asc --encrypt ${ARC_DEST} 1>&2 &>/dev/null && rm ${RUN_FILE} 2>&1&
do_sleep 1 1 ":"; rm ${ARC_DEST}
 
echo -e "${C1} __________________________________________________________________________ "
echo -e "|                                                                          |"
echo -e "|                 ${C4} COMPLETED SUCCESSFULLY ${C1}                                 |"
echo -e "${C1} __________________________________________________________________________ ${C0} \n\n"
 
cd $OLDPWD
 
exit $?

Encrypted WordPress / phpBB Backups originally appeared on AskApache.com

This simple unix shell script automatically creates backups of a specific folder at regular hourly, nightly, weekly, and monthly intervals. DreamHost has this feature for their main accounts but for those of us running on the new Private Servers they haven't set it up yet. The old perl script they use on the main accounts does not work with the higher security of the PS kernels and virtual server setup.

I just learned that using your dreamhost account for backups like this shell-script is a violation of the Terms of Service, so please don't use this method on DreamHost.

Why Automated Backups

I make a lot of mistakes while developing and hacking code for my sites and I've come to rely on having access to these backup versions so I can quickly revert. Contacting support to get access to them is a waste of everyones time compared to finding a solution, so I did. The DreamHost wiki has a couple of complicated but workable solutions for backups but not for PS users, and nothing this simple. Just another great feature of DreamHosts debian linux based web hosting.

My Shell Script

Instead of the usual method for copying directory trees using tar with fifo, pipes, or DreamHost's rsync and NFS method this script uses cpio which is much much faster and has a lot of cool options like saving m/a/c times and symlinks, and also being able to handle weird characters and file names.

mysnapshot.sh shell script

#!/bin/bash
#
# GNU Free Documentation License 1.2
# 06-11-08 - AskApache (www.askapache.com)
#
#
#    .mysnapshot
#    |-- hourly.0    (one hour ago)
#    |-- nightly.0   (one night ago)
#    |-- weekly.0    (one week ago)
#    `-- monthly.0   (one month ago)
#
 
### Source and Destination
source=$HOME/sites
dest=$HOME/.mysnapshot/${1:-hourly}.0/`basename $source`
 
### Make Nice - lower load
renice 19 -p $$ &>/dev/null
 
### Non-Absolute links, check source exists
cd $source || exit 1
 
### Hide errs, copy dirtree
find . -depth -print0 2>/dev/null | cpio -0admp $dest &>/dev/null
 
cd $OLDPWD
 
exit 0

mysnapshot.sh crontab entries

MAILTO=user@domain.com
# MY SNAPSHOTS
@hourly /home/user/scripts/mysnapshot.sh hourly &>/dev/null
@midnight /home/user/scripts/mysnapshot.sh nightly &>/dev/null
@weekly /home/user/scripts/mysnapshot.sh weekly &>/dev/null
@monthly /home/user/scripts/mysnapshot.sh monthly &>/dev/null

Just save the script to your server, chmod u+x, add the crontab entries, and you will have automated backups of any folder you want other than your HOME folder.

Stay tuned, I'm learning some really incredible stuff right now *if you run with open-source* and will be posting more tutorials soon about some really cool stuff.

Automated Folder Backup Shell-Script originally appeared on AskApache.com

I've recently relocated to the DreamHost private server setup, not for any reason other than its such a cool offer that I had to take advantage of it to learn more about it.

DreamHost Private Server

DreamHost PS uses Linux-VServer to give you your own "virtual machine", thereby protecting your CPU and RAM from all other users on your physical machine.

Linux-VServer

Linux-VServer provides virtualization for GNU/Linux systems. This is accomplished by kernel level isolation. It allows to run multiple virtual units at once. Those units are sufficiently isolated to guarantee the required security, but utilize available resources efficiently, as they run on the same kernel. This particular virtual server model is implemented through a combination of "security contexts", segmented routing, chroot, extended quotas and some other standard tools.

What Bothers Me about DreamHostPS

1. They don't have the skill/desire to let you run only certain sites/usernames on the PS, its all or nothing, which completely sucks!
2. It's so darn expensive, I spent more after the first 24 hours than I usually do in a MONTH
3. Although they say you still have the backups of your site, the .snapshot folder is no longer accessible and you have to manually contact support to ask for a backup! Ahh!
4. The environment on the shell doesn't have as much access to good stuff like using locate
5. Processes can get out of control and use up to much memory if you have a busy site like me, which results in the sending of 503's to everyone else!
6. Still uses NFS, the slowest thing in the world and not worth it now that backups aren't even accessible.. it can also cause some hard-to-detect issues with caching setups

What Thrills me about DreamHostPS

1. I have more access and control over my server, sites, memory, and files via SSH login
2. I can scale the CPU and Memory whenever I want, and just like the static IP's you only pay for the time you use, making experimenting affordable!
3. I can reboot the whole server at anytime! Sometimes useful if a script I'm experimenting with locks the whole server..
4. Only my shell accounts can access the server! Security is much stronger!
5. You can configure other webservers like lighttpd, nginx, and fnord to serve content on sites that are mostly static content!
6. I've only had it for 3 days, so I'm sure I'll find alot more awesome capabilities

Problems migrating to DreamHostPS

Here are some issues I experienced during migration and the solutions I've used.

During this process it is important to note how helpful the DreamHost Support Staff were in putting up with my sometimes overly technical and detailed support requests. Thanks John, Brian, and Robert R!

PHP and HTTPD processes hogging all memory

Unlike on shared hosting accounts, where DH technical people have set up a very robust system, it appears they are missing the expertise of a past employee or something because this new setup is not as robust.. YET!

For instance I started out my account CPU and memory at the MAX (2300 MB / 2300 Mhz) but my sites were all still taking forever to serve content, simply because instead of on the shared servers where user processes and HTTPD instances are more controlled, this account seems to not have very well-thought out limits on it. So if 100 people asked for a page on my site, this server loads up 100 HTTPD processes under dhapache user and loads up 100 processes for the custom-compiled php.cgi I am running. This sounds like a cool thing but in reality it takes up so much of my memory that my bash shell login under SSH runs out of memory and won't even let me do a simple ps, and it just keeps serving 503's to anyone else who requests something on my site. DH will have to fix this soon or someone will launch a DDOS attack that will cripple them, unless a googlebot does it first!

Solution I contacted support and received a very friendly and prompt reply suggesting a bad script and offering to setup a process watcher and killer, which I accepted. Eventually I located the problem to be an ErrorDocument 500 directive in my .htaccess that was pointing to a php file instead of a static .html

No crontabs or cronjobs

None of my crontab files were moved to my new server and in fact I was receiving permission denied just to access my crontab.

Solution I contacted support and they installed new crontabs for me and offered to copy my old ones.

Static IP Changed for site with non-DreamHost DNS

One site uses DNS from Network Solutions, so when my site was migrated and got a new static IP address, my site went down. It would have been nice and should be expected that in this situation DreamHost would alert you that the change is going to happen so you can update your DNS without your site going offline.

Solution Logged into my Network Solutions account and updated the DNS for my site to point to the new Static IP.

SSH Hosts, Authorized Keys Broken

Some of my sites and user accounts use passwordless SSH to make some things work, and all of these were made useless when I moved to my new private server.

Solution

1. Logged into my user accounts with SSH
2. Deleted the old files in folder .ssh
3. Created new keys and added them to other accounts
4. Logged in to new accounts to add to host files

Old Server and Static IP References in Site Files

I have some pretty technical and complex cgi's, .htaccess files, shell scripts run by cronjobs, php scripts, etc., on some sites and shell accounts, and many of my files contain code to the Static IP and/or dreamhost server, either for access control or for faster connects by connecting straight to an IP instead of having to perform a DNS lookup. So when both the Static IP's and dreamhost server changed it broke all my files.

Solution Basically I knew I had to search all of my files and replace the old IP with the new IP. I also had to search files relacing the old server with my new server. To make life simpler, I wrote a simple shell script that I run from my account while logged in using SSH that does this automatically with the added feature of asking me if I would like to make the replacement for each file it finds, which is nice because I don't want to replace this for old log files and misc stuff.

dreamhostps migration shell script

#!/bin/bash
# Version 1.0 by AskApache 5/29/2008
 
shopt -s extglob
renice 19 $$
 
OLDSERVER=208.113.183.103
NEWSERVER=208.113.134.190
 
FIXFILES=$(grep -R -l -i $OLDSERVER $HOME/!(Maildir|logs|backups|source|tmp|doit|php5|php526|ip_abuse) 2>/dev/null)
 
for thefile in ${FIXFILES[@]}; do
 if [ -f "$thefile" ]; then
  echo -e "\n\n\n\n"
  echo "___________________________________________________________________"
  echo "Name:  ${thefile}"
  echo "Type:  $(command file -b ${thefile})"
  echo "Size:  $(command du -hs ${thefile}|awk '{ print $1}')"
  echo "Matching Lines:"
  grep -i --color=auto $OLDSERVER $thefile
  echo -e "___________________________________________________________________\n"
  echo -en "Replace occurances of $OLDSERVER with $NEWSERVER? [y/N] " ; read -n 1 ans
  case "$ans" in
   n|N) echo -e "\nSKIPPING..."; ;;
   y|Y) echo -e "\nREPLACING..."
        cp $thefile $thefile.b1 &>/dev/null
        cat $thefile.b1 | sed "s/${OLDSERVER}/${NEWSERVER}/g" 1>$thefile
        rm $thefile.b1 &>/dev/null
        echo "DONE"; ;;
  esac
 fi
done
exit 0

Other Webservers Allowed on DreamHostPS

Apache webserver on DH is configured in process-based mode. This means that each process serves one simultaneous connection and uses significant amount of memory for that, thus limiting concurrent user count. You can download and install some lightweight web-servers like (lighttpd) or (nginx). These servers use async i/o and can handle large count of concurrent connections without consuming much RAM, especially when serving static files. But default port 80 is already occupied by apache, and you cannot change that. So, you can use another port for new lightweight server, but visitors will see ugly port number in address and such solution will not work for some corporate firewalls.

1. lighttpd - small memory footprint compared to other web-servers, effective management of the cpu-load, and advanced feature set
   • FastCGI
   • SCGI
   • Auth
   • Output-Compression
   • URL-Rewriting
   • many more
2. nginx - lightweight web server/reverse proxy and e-mail (IMAP/POP3) proxy
   • Handling of static files, index files and auto-indexing
   • Reverse proxy without caching, load balancing, and fault tolerance
   • SSL support
   • FastCGI support
   • Name- and IP-based virtual servers
   • FLV streaming
3. fnord - lightweight webserver written by Felix von Leitner. It aims to be a small, but extremely fast and secure webserver.
   • Small! (13k static Linux-x86 binary without CGI, 18k with CGI)
   • Fast! Uses mmap (and on Linux, sendfile)
   • connection keep-alive
   • el-cheapo virtual domains (similar to thttpd)
   • IPv6 support (through tcpserver)
   • CGI (through pipes, not temp files like Apache)
   • Content-Range (not the full specs, just a-b or a- byte ranges)
   • transparent content negotiation
   • Directory index generation

DreamHost PS Troubleshooting

1. Symptoms of an Overloaded DreamHost PS
2. Checking your Daily & Monthly Resource Usage
3. Changing your CPU & Memory Allocation
4. Rebooting your DreamHost PS
5. Overloaded DreamHost PS
6. Idle DreamHost PS
7. Changing web server
8. See also
9. External link

Symptoms of an Overloaded DreamHost PS

If you experience any of the following symptoms, most likely you will need to increase the resource allocation (CPU & memory) to your Private Server:

1. Out of memory
2. Internal server errors ("500" errors)
3. Killed scripts
4. Inability to log in ("ssh_exchange_identification: Connection closed by remote host")

Checking your Daily & Monthly Resource Usage

Check the daily & monthly usage graph in the Control Panel under (PRIVATE SERVERS > CPU/MEMORY) to see if your usage is going beyond your currently guaranteed CPU and memory allocation.

Changing your CPU & Memory Allocation

You can increase or decrease your resource allocation by moving the green slider in the Control Panel under (PRIVATE SERVERS > CPU/MEMORY). As you move it the amount of CPU and memory will update automatically, along with the rate you'll be charged for that setting. Once you're happy with the setting click on the "Change {servername}'s CPU / Memory Now!" button to push the change into place. It will take a short period of time for the setting change to be reflected. Typically no reboot is necessary.

Rebooting your DreamHost PS

If you have a problem you can try rebooting your Private Server yourself in the Control Panel under (PRIVATE SERVERS > REBOOT SERVER). No need to contact support for that! Isn't that cool?! It may temporarily fix things, but if it doesn't provide a long-term solution, you will need to increase the resources allocated to your Private Server by using the green slider as mentioned in the previous paragraph.

Overloaded DreamHost PS

The image above is what your daily usage graph might look if you've got your settings too low. Notice that the actual usage routinely exceeds the guaranteed amount of 150MHz/MB. This is bound to cause problems for your sites. To rectify the situation it's recommended that you increase your resource allocation. Then keep an eye on the graph for several hours, and test our sites to see if you've allocated enough resources for things to run smoothly. It is recommended that you start out by doubling your current resource allocation to see if it's enough. Once you've verified that things are running properly you can reduce your resource allocation to the point where your peaks just barely exceed what you have allocated. Of course you'll want to routinely monitor your usage and increase the resource allocation as your needs increase. It's best to over allocate then under allocate! You don't want to find out that you've under allocated by your visitors/customers complaining about your sites not working properly.

Note that you're only charged for the period of time that you have the slider in a particular position. So it's safe to experiment. In fact, we recommend it. You can increase or decrease your resource allocation at any time.

You will typically see Apache processes running on your server and appearing to consume all of the memory. This is generally not the case because Apache processes share a significant amount of memory between one another. Additionally, we automatically configure Apache to work well within the memory allocation of your PS server. It is still possible for a busy website to overwhelm a DreamHost PS server, but it is not generally the fault of the apache webserver itself.

Of course you may also want to try to reduce your load on the server as well so you can reduce the resource allocation and save some money.

Idle DreamHost PS

The image above is what your daily usage graph might look if you've got nothing running under your DreamHost PS. This just shows the overhead resource usage. You could host a lightly loaded web page with this resource allocation (provided it uses static content), but probably not much more. If you've got your resource allocation set to 150MHz/MB it is recommendended that you monitor the usage very often.

Some Content on this page included from this article by Author History from the DreamHost Wiki and is licensed under the GNU FDL.

Upgrading to DreamHost Private Servers originally appeared on AskApache.com

This bash shell script searches in a directory you specify for any files containing a string and replaces that string with another string. It's nice because it asks you if you would like to be prompted for each operation, or to run without user-input. I use this shell script to change static IP addresses or domains, file locations, database info, etc.

Note: I'm going to be updating this soon..

For now here is a one-liner for you that works very fast and is also low-resource-intensive!

nice -n19 sh -c 'S=askapache && R=htaccess; find . -type f|xargs -P5 -iFF grep -l -m1 "$S" FF|xargs -P5 -iFF sed -i -e s%${S}%${R}% FF'

ScreenShot

Shell Script Code

You can also download it: FIND AND REPLACE SCRIPT Version 1.4

#!/bin/bash
# Find and Replace Version 2, 2009-04-19
# Version 1.4
# GNU Free Documentation License 1.2
# 2009-04-19 - AskApache (www.askapache.com)
umask 0022
 
# set -o xtrace && set -o verbose #uncomment for verbose debugging
 
# nicify to avoid getting killed
renice 19 -p $$ &>/dev/null
 
### SHELL OPTIONS
set +o noclobber  # allowed to clobber files
set +o noglob     # globbing on
set -e            # abort on first error
shopt -s extglob  # more globbing
 
PATH=/bin:/sbin:/usr/bin:/usr/local/bin:/usr/libexec:/usr/sbin
VER="1.4"
COLORON=yes
 
SEARCH=${1:-QQQQQQQQ};
REPLACE=${2:-AskApache};
SDIR=${3:-/tmp};
 
# script_title
#==-==-==-==-==-==-==-==-==-==-==#
function script_title(){
  clear;echo -e "${C[1]} __________________________________________________________________________ "
  echo -e "| ${C[2]}             ___       __    ___                 __             ${C[1]}         |"
  echo -e "| ${C[2]}            / _ | ___ / /__ / _ | ___  ___ _____/ /  ___        ${C[1]}         |"
  echo -e "| ${C[2]}           / __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)       ${C[1]}         |"
  echo -e "| ${C[3]}          /_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/        ${C[1]}         |"
  echo -e "| ${C[3]}                               /_/                              ${C[1]}         |"
  echo -e "|                                                                          |"
  echo -e "|                 ${C[4]} FIND AND REPLACE SCRIPT Version $VER ${C[1]}                    |"
  echo -e "${C[1]} __________________________________________________________________________ ${C[0]} \n\n"
}
 
# pm
#==-==-==-==-==-==-==-==-==-==-==#
function pm()
{ case ${2:-d} in d) echo -en "\n\n${C[2]}>>> ${C[4]}$1 ${C[0]}\n\n"; ;; i*) echo -e "\n${C[6]}=> ${C[4]}${1} ${C[0]}\n"; ;; esac; return 0; }
 
# ok_continue
#==-==-==-==-==-==-==-==-==-==-==#
function ok_continue()
{ local ans; echo -en "${C[4]} \n [ Press any key to continue ] ${C[0]} \n"; read -n 1 ans; return 0; }
 
# yes_no
#==-==-==-==-==-==-==-==-==-==-==#
function yes_no()
{ local a YN=65; echo -en "${C[2]}>>> ${C[4]}${1:-Q} [y/n] ${C[0]}"; read -n 1 a; case $a in [yY]) YN=0; ;; esac; return $YN; }
 
# p_done
#==-==-==-==-==-==-==-==-==-==-==#
function p_done()
{ echo -e "\n${C[8]} DONE ${C[0]} \n"; return 0; }
 
# get_settings
#==-==-==-==-==-==-==-==-==-==-==#
function get_settings(){
  local a cha
  script_title
  while true; do
  for a in "SDIR" "SEARCH" "REPLACE"; do
   cha=g; echo -en "\n ${C[4]}(Enter for Default: ${!a} )${C[0]}\n ${a}${C[6]}=> ${C[0]} ";
   read -e cha; echo; [[ ${#cha} -gt 2 ]] && eval "$a"=$cha;
  done
  yes_no "ARE THESE SETTINGS CORRECT" && echo -e "\n\n" && break
  done
}
 
# search_and_replace
#==-==-==-==-==-==-==-==-==-==-==#
function search_and_replace(){
  for i in $(seq 0 $((${#FOUNDFILES[@]} - 1))); do
  thefile=${FOUNDFILES[$i]}
 
  if [[ $1 -eq 0 ]]; then
   echo -e "\n\n\n___________________________________________________________________\n"
   echo -e "${C[4]}FILE:  ${C[3]}${thefile} ${C[2]}($(command du -hs ${thefile}|awk '{ print $1}'))"
   echo -e "${C[4]}Type:  ${C[6]}$(command file -b ${thefile})\n${C[7]}Matching Lines:${C[0]}"
   command grep --color=always $SEARCH $thefile;echo
 
   while true; do
    yes_no "Replace occurances of ${SEARCH} with ${REPLACE}?" && echo -e "\nREPLACING...\n" && sed -i -e "s%${SEARCH}%${REPLACE}%g" $thefile && p_done && break
    echo -e "\nSKIPPING..\n" && p_done && break
   done
 
   else
    sed -i -e "s%${SEARCH}%${REPLACE}%g" $thefile
   fi
 
  done
  return 0;
}
 
############################################################################################################
### MAIN CODE
############################################################################################################
 
if [[ $SEARCH == "QQQQQQQQ" ]]; then
  E='\033[';
  [[ -z "$TERM" ]]&&[[ -z "$1" ]]||declare -a C=('${E}0m' '${E}1\;30m' '${E}0\;32m' '${E}1\;32m' '${E}1\;37m' '${E}1\;36m' '${E}1\;35m' '${E}1\;37m' '${E}30\;42m' '${E}1\;36m' )
  script_title            # DISPLAY SCRIPT TITLE
 
  get_settings            # GET SCRIPT SETTINGS
  cd $SDIR
  declare -a FOUNDFILES=( `command grep -R -l $SEARCH . 2>/dev/null` );
  declare -a FOUNDMATCHES=(  );
  for i in $(seq 0 $((${#FOUNDFILES[@]} - 1))); do F=${FOUNDFILES[$i]}; FOUNDMATCHES[$i]=$(command grep -c . $F); done
  pm "FOUND ${SEARCH} IN ${#FOUNDFILES[@]} FILES" i
  for i in $(seq 0 $((${#FOUNDFILES[@]} - 1))); do echo -e "${C[4]}${FOUNDFILES[$i]} ${C[6]}=>${C[4]} ${FOUNDMATCHES[$i]} matches"; done
  echo -e "\n"
 
 while true; do
  yes_no "Replace all occurances of ${SEARCH} with ${REPLACE} without prompting?" && search_and_replace 1 && p_done && break
  search_and_replace 0 && p_done && break
 done
 
else
 cd $SDIR
 command grep -H -R -l $SEARCH . 2>/dev/null | xargs -iFFF sed -i -e "s%${SEARCH}%${REPLACE}%g" FFF
fi
 
exit 0

Simplified for crontab

*Mike*
To run from a crontab you wouldn't want a full-fledged script like this, I would just create a small bash script like so:

#!/bin/sh
PATH=/usr/bin:$PATH:/usr/local/bin
command grep -H -R -l $1 $3 2>/dev/null | xargs -iFFF sed -i -e "s%${1}%${$2}%g" FFF

This is like 100x faster, to call from crontab every 35 minutes and search the directory /home/user/tmp for 'search' and replace it with 'replace' every 35 minutes.

*/35 * * * * /bin/sh /home/user/search-replace.sh 'search' 'replace' '/home/user/tmp' 1>/dev/null

Search And Replace shell script helpful for Upgrades originally appeared on AskApache.com