Note: I spent no time on readability, this is primarily a read the code and figure it out article.. This is for advanced users looking for a reference or discussion and for those of you looking to advance. Feedback would be great if you make it that far..

For a better handle on the way I like to structure web site directories, see Optimize a Website for Speed, Security, and Easy Management but note it is a bit outdated compared to what I’m doing now. I don’t have the luxury of using only one type of server, or hosting provider anymore, so I have been working towards making things even more portable in order to move from host to host from server to server without issues i.e. my portable .bash_profile.

So I’ve been basically experimenting various ways to accomplish that and thought I would share what I am currently doing for my benefit and hopefully get some input. All of my WP installs run the development version, and one main idea with my setups is that upgrading is automated. So I really keep the WordPress install clean and use plugins and wp-config.php to do all the customization.

• Portability – Hands-free upgrades and easy to move
• Security – Additional security and protection
• Speed – Less CPU and Disk I/O
• Customization – All my favorite customizations

wp-config.php

These are the main settings I use.. Seriously this is more like an interactive article, because to understand it you will need to do some code grepping. You may want to grab a jolt.

ASKAPACHE_ROOT

The ASKAPACHE_ROOT variable is just a better way for me to be able to include and access all the different files in my site tree. For instance, in my non-wp php files, I can do this:

!defined('ASKAPACHE_ROOT') && require $_SERVER['DOCUMENT_ROOT'] . '/wp-config.php';
include(ASKAPACHE_ROOT . '/includes/custom-download.inc.php');

ASKAPACHE_LOCK

This is one of my all-time favorite hacks, that I think is one of the most useful methods I employ as a web developer. This allows me to use far-future-expire headers for optimum caching, while still forcing browsers to re-validate every day or so automatically, or forcing them to re-validate whenever I change the suffix. This takes advantage of the mod_rewrite trick that I use on EVERY site I run, definately worth learning. Because I practice best-practice web-standards, for every web site I create a single css file and javascript file, which I then add to the template like:

<link rel="stylesheet" type="text/css" media="all" href="http://static.askapache.com/c/apache-0<?php echo ASKAPACHE_LOCK?>.css" />
<script src="http://static.askapache.com/j/apache-0<?php echo ASKAPACHE_LOCK;?>.js" type="text/javascript"></script>

<?php
/**
 * The base configurations of the WordPress.
 *
 * This file has the following configurations: MySQL settings, Table Prefix,
 * Secret Keys, WordPress Language, and ABSPATH. You can find more information by
 * visiting {@link http://codex.wordpress.org/Editing_wp-config.php Editing
 * wp-config.php} Codex page. You can get the MySQL settings from your web host.
 *
 * This file is used by the wp-config.php creation script during the
 * installation. You don't have to use the web site, you can just copy this file
 * to "wp-config.php" and fill in the values.
 *
 * @package WordPress
 */
/* http://codex.wordpress.org/Editing_wp-config.php */
 
/** /home/liet/askapache.com */
!defined('ASKAPACHE_ROOT') && define('ASKAPACHE_ROOT', str_replace('/public_html','', $_SERVER['DOCUMENT_ROOT']));
 
/** The 008 at the end is for manual tweaking.  time() returns seconds since '00:00:00 1970-01-01 UTC'. */
// http://www.askapache.com/htaccess/mod_rewrite-fix-for-caching-updated-files.html
!defined('ASKAPACHE_LOCK') && define(ASKAPACHE_LOCK', substr(time(),0,5).'008'); // 12533001
 
/** absolute path to the WordPress directory */
!defined('ABSPATH') && define('ABSPATH', ASKAPACHE_ROOT .'/public_html/');
 
/**
 * WP_SITEURL, defined since WordPress Version 2.2, allows the WordPress address (URL) to be defined. The valued defined is the address where your WordPress core files reside.
 * It should include the http:// part too. Do not put a slash "/" at the end.
 * Setting this value in wp-config.php overrides the wp_options table value for siteurl and disables the WordPress address (URL) field in the Administration > Settings > General panel.
 */
!defined('WP_SITEURL') && define('WP_SITEURL', 'http://'.$_SERVER['SERVER_NAME']);
 
/**
 * WP_HOME is another wp-config.php option added in WordPress Version 2.2. Similar to WP_SITEURL,
 * WP_HOME overrides the wp_options table value for home but does not change it permanently.
 * home is the address you want people to type in their browser to reach your WordPress blog. It should include the http:// part. Also, do not put a slash "/" at the end.
 */
!defined('WP_HOME') && define('WP_HOME', WP_SITEURL);
 
/** no trailing slash, full paths only */
!defined('WP_CONTENT_DIR') && define( 'WP_CONTENT_DIR', ABSPATH . 'wp-content' );
 
// full url - WP_CONTENT_DIR is defined further up
!defined('WP_CONTENT_URL') && define( 'WP_CONTENT_URL', WP_SITEURL . '/wp-content');
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full path, no trailing slash
!defined('WP_PLUGIN_DIR') && define( 'WP_PLUGIN_DIR', WP_CONTENT_DIR . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.6.0 */
// full url, no trailing slash
!defined('WP_PLUGIN_URL') && define( 'WP_PLUGIN_URL', WP_CONTENT_URL . '/plugins' );
 
/** Allows for the plugins directory to be moved from the default location. @since 2.1.0 */
// Relative to ABSPATH.  For back compat.
//!defined('PLUGINDIR') && define( 'PLUGINDIR', 'wp-content/plugins' );
 
/** Number of autosaves to save. TRUE is default and enables post revisions, FALSE disables revisions completely. */
!defined('WP_POST_REVISIONS') && define('WP_POST_REVISIONS', 150);
 
/* ini_set('memory_limit', WP_MEMORY_LIMIT); */
!defined('WP_MEMORY_LIMIT') && define('WP_MEMORY_LIMIT', '64M');
 
/** Only check at this interval for new messages. Default is 5min */
/** @since 2.9  */
!defined('WP_MAIL_INTERVAL') && define('WP_MAIL_INTERVAL', 3600); // 1 hour
 
/** Saves updated post values to post from edit window every x seconds. (default 60)
 * When editing a post, WordPress uses Ajax to auto-save revisions to the post as you edit. You may want to increase this setting for longer delays in between auto-saves, or decrease the setting to make sure you never lose changes.
 * @since 2.5.0 */
!defined( 'AUTOSAVE_INTERVAL' ) && define( 'AUTOSAVE_INTERVAL', 60 );
 
/** @since 2.9.0  */
/** Permanently deletes posts, pages, attachments, and comments which have been in the trash for EMPTY_TRASH_DAYS. */
!defined( 'EMPTY_TRASH_DAYS' ) && define( 'EMPTY_TRASH_DAYS', 300 );

Debugging WordPress

One of my secrets for getting really good at this stuff is to master debugging. There is really not ever a time when I am working on a site that I don’t have color-highlighted logs scrolling automatically in an ssh window. It’s really almost impossible to fix problems with wordpress or do any kind of advanced anything without being able to view debugging info. At first I relied heavily on a custom php.ini being available on the server, but after having to deal with many hosts who don’t allow php.ini files I now rely completely on setting values using ini_set for ultimate portability. Detailed towards the end of this article and is also included in this wp-config.php

/**#@+
 * DEBUGGING STUFF
 */
/** display of notices during development. if false, error_reporting is E_ERROR | E_WARNING | E_PARSE | E_USER_ERROR | E_USER_WARNING | E_RECOVERABLE_ERROR otherwise E_ALL */
!defined('WP_DEBUG') && define('WP_DEBUG', false);
 
/** The SAVEQUERIES definition saves the database queries to a array and that array can be displayed to help analyze those queries.
 *  The information saves each query, what function called it, and how long that query took to execute.  */
!defined('SAVE_QUERIES') && define('SAVE_QUERIES', WP_DEBUG);
 
!defined('ACTION_DEBUG') && define('ACTION_DEBUG', WP_DEBUG);
 
/** This will allow you to edit the scriptname.dev.js files in the wp-includes/js and wp-admin/js directories.  */
!defined('SCRIPT_DEBUG') && define('SCRIPT_DEBUG', WP_DEBUG);

 
/** Add define('WP_DEBUG_LOG', true); to enable php debug logging to WP_CONTENT_DIR/debug.log */
//!defined('WP_DEBUG_LOG') && define('WP_DEBUG_LOG', true);
 
/** This determines whether errors should be printed to the screen as part of the output or if they should be hidden from the user.
 *  Add define('WP_DEBUG_DISPLAY', false); to wp-config.php to use the globally configured setting for display_errors and not force it to On */
!defined('WP_DEBUG_DISPLAY') && define('WP_DEBUG_DISPLAY', false);

Ultimate Security Tweaks

Well, ultimate for WP’s built-in keys and password functions, this is all for wp-config.php keep in mind. This is a very neccessary and recommended step, and is one of the only things I modify for each new installation.

Security KEYS

If like me you are familiar with password-cracking software like John the ripper, rainbow hash tables, l0pht-crack, etc.. then you will like to know that you can specify your own keys and salts for the encryption used by WP. They are AUTH_KEY, AUTH_SALT, SECURE_AUTH_KEY, SECURE_AUTH_SALT, LOGGED_IN_KEY, LOGGED_IN_SALT, NONCE_KEY, NONCE_SALT, SECRET_KEY and SECRET_SALT.

A random and long key gives you better encryption, and exponentially increasing that is using a random and long salt for the encryption. Encryptions with known salts are incredibly easy to decrypt compared to encryptions with secure salts, because the salt + key individually need to be guessed in order to find a matching hash, vs. just the key if the salt is known. See: Locating weak passwords.

A secret key is a hashing salt which makes your site harder to hack and access harder to crack by adding random elements to the password.

In simple terms, a secret key is a password with elements that make it harder to generate enough options to break through your security barriers. A password like “password” or “test” is simple and easily broken. A random, unpredictable password such as “88a7da62429ba6ad3cb3c76a09641fc” takes years to come up with the right combination.

For more information on the technical background and breakdown of secret keys and secure passwords, see:

• WordPress Support Forum – HOWTO: Set up secret keys in WordPress 2.6+
• Wikipedia’s explanation of Password Cracking

I like to use the WordPress.org secret-key service 4 times. That’s because for each key and salt I like to do: (1 key from api +random keyboard input+1 key from api).

/**#@+
 * Authentication Unique Keys.
 *
 * Change these to different unique phrases!
 * You can generate these using the {@link https://api.wordpress.org/secret-key/1.1/ WordPress.org secret-key service}
 * You can change these at any point in time to invalidate all existing cookies.
 * This will force all users to have to log in again.
 *
 * @since 2.6.0
 *
 * Get salt to add to hashes to help prevent attacks.
 *
 * The secret key is located in two places: the database in case the secret key
 * isn't defined in the second place, which is in the wp-config.php file. If you
 * are going to set the secret key, then you must do so in the wp-config.php
 * file.
 *
 * The secret key in the database is randomly generated and will be appended to
 * the secret key that is in wp-config.php file in some instances. It is
 * important to have the secret key defined or changed in wp-config.php.
 *
 * If you have installed WordPress 2.5 or later, then you will have the
 * SECRET_KEY defined in the wp-config.php already. You will want to change the
 * value in it because hackers will know what it is. If you have upgraded to
 * WordPress 2.5 or later version from a version before WordPress 2.5, then you
 * should add the constant to your wp-config.php file.
 *
 * Below is an example of how the SECRET_KEY constant is defined with a value.
 * You must not copy the below example and paste into your wp-config.php. If you
 * need an example, then you can have a
 * {@link https://api.wordpress.org/secret-key/1.1/ secret key created} for you.
 *
 * Salting passwords helps against tools which has stored hashed values of
 * common dictionary strings. The added values makes it harder to crack if given
 * salt string is not weak.
 *
 * @since 2.5
 * @link https://api.wordpress.org/secret-key/1.1/ Create a Secret Key for wp-config.php
 *
 * @return string Salt value from either 'SECRET_KEY' or 'secret' option
 */
define('AUTH_KEY',        'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?p[B+GR{@>{Yq`c|LnG;dvq#| %OA_cbBSU6,rICC1o/c)-|');
define('SECURE_AUTH_KEY', 'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bb15baar8&R-r<[T|?(xhJJABGq+Ux+U$)-Hltp/');
define('LOGGED_IN_KEY',   'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B<5n6DG|YWnJ9tY2!M1L)`{-$LW~~Ia%.uCbn!P. 41o2$Z$4');
define('NONCE_KEY',       'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[Bgu<wM*zewR0.{+m:bmrB?wj!B,4]Wo+4 Avk ApR-D?E');
define('SECRET_KEY',     'jflkhaskljdfhkljasdhflkjashd;flkjhas;djfh;kajshdflkjashdlfkjhasdlkfhal?Vp[B52ugH6muE9r4._iZwoYKUybrqLPpv|d Xr+|yrqhUE');
 
define('AUTH_SALT',        '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm~Ky%+%~PPa5b YEmDI%U[W!-B');
define('SECURE_AUTH_SALT', '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmad/7o6.AU3%9o-|Kqm]+eUqr-n~:ag');
define('LOGGED_IN_SALT',   '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmsLiCv@KJ{#wd(?qe(KcH3!');
define('NONCE_SALT',       '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdmG9>+wm 2)bS0Pd_+1rx0brX]ND8|');
define('SECRET_SALT',      '123423190847olqkfhladhfsldshafasdfasdf09a7f-90a87df98adfyapoiyaf9asd8f70a9s8d7f908a7sdf97W4qCdm2<>))U|sty)+4vpWooKls/^[vN');
/**#@-*/

Using SSL for Admin and Login

SSL is kinda required from my point of view, it is just way to easy to sniff data off the wire otherwise. At least with SSL you force them to use tools like burpsuite, paros proxy, webscarab, etc..

/** @since 2.6.0  */
!defined('FORCE_SSL_ADMIN') && define('FORCE_SSL_ADMIN', true);
 
/** @since 2.6.0  */
!defined('FORCE_SSL_LOGIN') && define('FORCE_SSL_LOGIN', true);

Mod_Rewrite to Force SSL

This is pretty cool, it forces non-https for all urls except for /wp-admin and wp-login.php, which both require https. It also checks for the logged_in_cookie, and if that is present in the request then it doesn't force non-https. Kinda confusing if you don't have a mod_rewrite cheatsheet.

RewriteCond %{THE_REQUEST} ^$ [OR]
RewriteCond %{REQUEST_URI} ^/(wp-admin|wp-login\.php).*$ [NC,OR]
RewriteCond %{HTTP_COOKIE} ^.*wp_li_sadfsdfasdf11b361cdsdfasdfasd=.*$ [NC]
RewriteRule .* - [S=1]
 
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule .* http://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
 
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(wp-admin/.*|wp-login\.php.*)\ HTTP/ [NC]
RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

File System Permissions

You can get a basic and solid intro on file permissions by reading: Changing File Permissions, or you can check out some of my file permission research.

/** The permissions as octal number, usually 0644 for files, 0755 for dirs.
 *  http://codex.wordpress.org/Changing_File_Permissions
 *  if ( !$wp_filesystem->mkdir($remote_destination, FS_CHMOD_DIR) )
 */
!defined('FS_CHMOD_DIR') && define('FS_CHMOD_DIR', (0755 & ~ umask()));
!defined('FS_CHMOD_FILE') && define('FS_CHMOD_FILE', (0644 & ~ umask()));
/**#@-*/
 
/** Define the timeouts for the connections. Only available after the construct is called to allow for per-transport overriding of the default. */
//stream_set_timeout( $stream, FS_TIMEOUT );
//!defined('FS_TIMEOUT') && define('FS_TIMEOUT', 30);
 
//$this->link = @ftp_connect($this->options['hostname'], $this->options['port'], FS_CONNECT_TIMEOUT);
//!defined('FS_CONNECT_TIMEOUT') && define('FS_CONNECT_TIMEOUT', 30);
 
// function get_filesystem_method($args = array(), $context = false) {
//  $method = defined('FS_METHOD') ? FS_METHOD : false; //Please ensure that this is either 'direct', 'ssh', 'ftpext' or 'ftpsockets'
//!defined('FS_METHOD') && define('FS_METHOD', 'direct');
 
/** These methods for the WordPress core, plugin, and theme upgrades try to determine the WordPress path, as reported by PHP, but symlink trickery can sometimes
 * 'muck this up' so if you know the paths to the various folders on the server, as seen via your FTP user, you can manually define them in the wp-config.php file.
 * FS_METHOD forces the filesystem method. It should only be "direct", "ssh", "ftpext", or "ftpsockets".
 * FTP_BASE is the full path to the "base" folder of the WordPress installation.
 * FTP_CONTENT_DIR is the full path to the wp-content folder of the WordPress installation.
 * FTP_PLUGIN_DIR is the full path to the plugins folder of the WordPress installation.
 * FTP_PUBKEY is the full path to your SSH public key.
 * FTP_PRIKEY is the full path to your SSH private key.
 * FTP_USER is either user FTP or SSH username. Most likely these are the same, but use the appropriate one for the type of update you wish to do.
 * FTP_PASS is the password for the username entered for FTP_USER. If you are using SSH public key authentication this can be omitted.
 * FTP_HOST is the hostname:port combination for your SSH/FTP server. The standard FTP port is 21 and the standard SSH port is 22.
 */
//define('FS_METHOD', 'ftpext');
//define('FTP_BASE', '/path/to/wordpress/');
//define('FTP_CONTENT_DIR', '/path/to/wordpress/wp-content/');
//define('FTP_PLUGIN_DIR ', '/path/to/wordpress/wp-content/plugins/');
//define('FTP_PUBKEY', '/home/username/.ssh/id_rsa.pub');
//define('FTP_PRIKEY', '/home/username/.ssh/id_rsa');
//define('FTP_USER', 'username');
//define('FTP_PASS', 'password');
//define('FTP_HOST', 'ftp.example.org:21');
 
/**
 * Block requests through the proxy.
 *
 * Those who are behind a proxy and want to prevent access to certain hosts may do so. This will
 * prevent plugins from working and core functionality, if you don't include api.wordpress.org.
 *
 * You block external URL requests by defining WP_HTTP_BLOCK_EXTERNAL in your wp-config.php file
 * and this will only allow localhost and your blog to make requests.
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
/** @since 2.9  */
//!defined('WP_HTTP_BLOCK_EXTERNAL') && define( 'WP_HTTP_BLOCK_EXTERNAL', false );
 
/*
 * The constant WP_ACCESSIBLE_HOSTS will allow additional hosts to go through for requests. The format of the
 * WP_ACCESSIBLE_HOSTS constant is a comma separated list of hostnames to allow.
 *
 * @since 2.8.0
 * @link http://core.trac.wordpress.org/ticket/8927 Allow preventing external requests.
 * $accessible_hosts = preg_split('|,\s*|', WP_ACCESSIBLE_HOSTS);
 * return !in_array( $check['host'], $accessible_hosts ); //Inverse logic, If its in the array, then we can't access it.
 */
//!defined('WP_ACCESSIBLE_HOSTS') && define( 'WP_ACCESSIBLE_HOSTS', 'askapache.com,askapache.org' );

Cookies!

There’s always a little comfort in having non-default cookies for security (against auto-bots), and using shorter names also means smaller HTTP Packets.

The $cookie_hash is my hack to get around the fact that COOKIEHASH isn’t definable in wp-config.

/**#@+
 * COOKIES
 * Used to guarantee unique hash cookies @since 1.5 */
$cookie_hash=md5(WP_SITEURL);
 
/** Set a cookie now to see if they are supported by the browser.
 * setcookie(TEST_COOKIE, 'WP Cookie check', 0, COOKIEPATH, COOKIE_DOMAIN);
 * @since 2.3.0 */
!defined('TEST_COOKIE') && define('TEST_COOKIE', 'wp_tc');
 
/* @since 2.6.0 */
!defined('LOGGED_IN_COOKIE') && define('LOGGED_IN_COOKIE', 'wp_li_' . $cookie_hash);
 
/* @since 2.6.0 */
!defined('SECURE_AUTH_COOKIE') && define('SECURE_AUTH_COOKIE', 'wp_sa_' . $cookie_hash);
 
/* @since 2.5.0 */
!defined('AUTH_COOKIE') && define('AUTH_COOKIE', 'wp_a_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('PASS_COOKIE') && define('PASS_COOKIE', 'wp_p_' . $cookie_hash);
 
/* @since 2.0.0 */
!defined('USER_COOKIE') && define('USER_COOKIE', 'wp_u_' . $cookie_hash);
 
/* ok unset this var, its not needed as COOKIEHASH will have this value, but is not definable in wp-config.php */
unset($cookie_hash);
 
/** @since 1.2.0 */
!defined('COOKIEPATH') && define('COOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_HOME . '/' ) );
 
/** @since 1.5.0 */
!defined('SITECOOKIEPATH') && define('SITECOOKIEPATH', preg_replace('|https?://[^/]+|i', '', WP_SITEURL . '/' ) );
 
/** @since 2.6.0 */
!defined('ADMIN_COOKIE_PATH') && define( 'ADMIN_COOKIE_PATH', SITECOOKIEPATH . 'wp-admin' );
 
/** @since 2.6.0 */
!defined('PLUGINS_COOKIE_PATH') && define( 'PLUGINS_COOKIE_PATH', preg_replace('|https?://[^/]+|i', '', WP_PLUGIN_URL)  );
 
/** @since 2.0.0 */
!defined('COOKIE_DOMAIN') && define('COOKIE_DOMAIN', $_SERVER['SERVER_NAME']);

/**
  * The WP_CACHE setting, if true, includes the wp-content/advanced-cache.php script, when executing wp-settings.php.
  * For an advanced caching plugin to use, static because you would only want one
  * if ( defined('WP_CACHE') )@include WP_CONTENT_DIR . '/advanced-cache.php';
  */
!defined('WP_CACHE') && define('WP_CACHE', true);
 
/** WordPress Localized Language, defaults to en_US.
 *
 * Change this to localize WordPress.  A corresponding MO file for the chosen
 * language must be installed to wp-content/languages. For example, install
 * de.mo to wp-content/languages and set WPLANG to 'de' to enable German
 * language support. */
!defined('WPLANG') && define ('WPLANG', 'en_US');
 
/** Stores the location of the language directory. First looks for language folder in WP_CONTENT_DIR
 *   and uses that folder if it exists. Or it uses the "languages" folder in WPINC. @since 2.1.0 */
//!defined('WP_LANG_DIR') && define('WP_LANG_DIR', ABSPATH . WPINC . '/languages');
 
/** LANGDIR defines what directory the WPLANG .mo file resides. If LANGDIR is not defined WordPress looks first to wp-content/languages and then wp-includes/languages for the .mo defined by WPLANG file.  Old static relative path maintained for limited backwards compatibility - won't work in some cases*/
//!defined('LANGDIR') && define('LANGDIR', 'wp-content/languages');
 
/** Stores the location of the WordPress directory of functions, classes, and core content. @since 1.0.0 */
//!defined('WPINC') && define('WPINC', 'wp-includes');

WPMU Stuff

I personally don’t use.

/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_DIR') && define( 'WPMU_PLUGIN_DIR', WP_CONTENT_DIR . '/mu-plugins' ); // full path, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined('WPMU_PLUGIN_URL') && define( 'WPMU_PLUGIN_URL', WP_CONTENT_URL . '/mu-plugins' ); // full url, no trailing slash
 
/** Allows for the mu-plugins directory to be moved from the default location. @since 2.8.0 */
//!defined( 'MUPLUGINDIR' ) && define( 'MUPLUGINDIR', 'wp-content/mu-plugins' ); // Relative to ABSPATH.  For back compat.

WordPress Database

This is usually the only thing I have to manually edit when creating a new site, unless I just use the same DB and modify the $table_prefix, (farther down). I run everything I possibly can in UTF-8, but if you don’t already know alot about character sets, wow it is one of the most confusing things so you may want to save learning about that topic for another day. Otherwise the following are helpful (and show how confusing character sets are!)

• Character Sets and Collations MySQL Support
• Converting Database Character Sets
• UTF-8 character sets (UTF-8)

If you ever setup WP to use the builtin membership features, make sure you learn about the CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE constants, I’ve found them very helpful.

/**#@+
 * MySQL settings
 */
/** The name of the database for WordPress */
define('DB_NAME', 'askapachewpblog75');
 
/** The username to access the database */
define('DB_USER', 'askapache245d');
 
/** The password for the username to access the database */
define('DB_PASSWORD', 'asdfklj2340');
 
/** The hostname to connect to the database at */
define('DB_HOST', 'mysql.askapache.com');
 
/** The charset of the database */
define('DB_CHARSET', 'utf8');
 
/** The collation of the database */
define('DB_COLLATE', 'utf8_general_ci');

$table_prefix

The $table_prefix is the value placed in the front of your database tables. Change the value if you want to use something other than wp_ for your database prefix. Typically this is changed if you are installing multiple WordPress blogs in the same database, and also for enhanced security.

Its a safe and good idea to change this value pre-installation to add more security to your WordPress blog. Exploits attempted against your WordPress blog by malicious crackers often are built with the premise that your blog uses the prefix wp_, by changing the value you mitigate some attack vectors.

/**
 * WordPress Database Table prefix.
 *
 * You can have multiple installations in one database if you give each a unique
 * prefix. Only numbers, letters, and underscores please!
 */
$table_prefix  = 'ar15_';
 
/** CUSTOM_USER_TABLE and CUSTOM_USER_META_TABLE are used to designated that the user and usermeta tables normally utilized by WordPress are not used, instead these values/tables are used to store your user information. */
//!defined('CUSTOM_USER_TABLE') && define('CUSTOM_USER_TABLE', $table_prefix . 'my_users');
//!defined('CUSTOM_USER_META_TABLE') && define('CUSTOM_USER_META_TABLE', $table_prefix . 'my_usermeta');

Setup PHP Ini Settings

 
/** Turns the output of errors on or off, you really never want this on, you should only view errors by reading the log file. */
ini_set('display_errors', WP_DEBUG_DISPLAY);
 
/** Tells whether script error messages should be logged to the server's error log or error_log. */
ini_set('log_errors', 'On');
 
/** http://us.php.net/manual/en/timezones.php */
ini_set('date.timezone', 'America/Indianapolis');
 
/** Where to log php errors */
ini_set('error_log', ASKAPACHE_ROOT . '/logs/php_error.log');
 
/** Set the memory limit, otherwise defaults to '32M' */
ini_set('memory_limit', WP_MEMORY_LIMIT);

Sessions are slow

So I only use sessions when I have a specific use… In this case I need sessions only when one of the tools in the /online-tools/ directory is being used. And that is for the captcha image. In the future I won’t ever use sessions.

if(preg_match( '#^/online-tools/#',$_SERVER['REQUEST_URI'])) session_start();

Include Custom Files

Sure you could use the my-hacks.php that WP allows, or you can just stick your functions in your TEMPLATEPATH/functions.php file, but they are executed only after the wp-settings.php file, which may be too late for your file.

In the past I’ve also used the auto_prepend_file settings to run my script before anything (index.php) but I ran into some issues on different hosts, and it wasn’t as portable.

This is useful because you can have a file with globally available functions that you can use in non-WP areas as well as WP areas. I am moving away from this more and more as I learn more about classes and build plugins instead for portability.

include_once ASKAPACHE_ROOT . '/includes/myfunctions.inc';
 
/** Sets up WordPress vars and included files. */
require_once(ABSPATH . 'wp-settings.php');
?>

Some Useful PHP

I am constantly trying to make my sites and code more portable, so I am using plugins alot more to accomplish things that I use to do with separate php. Here are some examples of minimal php.

add_filter("the_generator", create_function('$a','return "";'));
add_filter('the_content', create_function('$a', 'return ((is_feed())? $a."<p><a href=\"".get_permalink()."\">".get_the_title()."</a> originally appeared on ".get_bloginfo("name").".</p>" : $a);'), 99999);
add_filter('excerpt_length', create_function('$a', 'return 300;'),99);
add_filter('excerpt_more', create_function('$a', 'return "&hellip;";'),99);
add_action( 'wp_head', create_function('$a','echo "<link rel=\"pingback\" href=\"'.get_bloginfo('pingback_url').'\" />\n";'), 95 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rss\" href=\"http://purl.org/rss/1.0/\" />\n";'), 96 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"schema.rel\" href=\"http://purl.org/vocab/relationship/\" />\n";'), 97 );
add_action( 'wp_head', create_function('$a','echo "<link rel=\"meta\" type=\"application/rdf+xml\" href=\"/foaf.rdf\" />\n";'), 98 );
add_action( 'wp_head', create_function('$a','echo "<link href=\"/favicon.ico\" rel=\"shortcut icon\" type=\"image/x-icon\" />\n";'), 99 );

Debugging Note

If you read this far than you probably know how important debugging is, but I sometimes like to stick the best tips deep in my articles to make sure only YOU find it. GRTFM isn’t used on this site, it’s mostly a requirement because my writing can get pretty bad.. The point, debugging is more than a crucial requirement if you want to do anything cool. Don’t worry I got you.. check my AskApache Debug Viewer Plugin from the official WP site. It’s pretty close to providing as verbose amount of information that I could possibly figure out how to get out of php, probably more than you have ever seen at least, I focused on quantity. I use it all the time on new installs as there is no setup required and it tells me advanced information about the setup of the server, hacker code for sure.

Here’s a quick function to see set global vars, I just think this is interesting code.

function askapache_global_debug(){
  global $_GET,$_POST,$_COOKIE,$_SESSION,$_ENV,$_FILES,$_SERVER,$_REQUEST,$HTTP_POST_FILES,$HTTP_POST_VARS,$HTTP_SERVER_VARS,$HTTP_RAW_POST_DATA,$HTTP_GET_VARS,$HTTP_COOKIE_VARS,$HTTP_ENV_VARS;
  $gv=create_function('$n','global $$n; ob_start(); if ( is_array($$n) && sizeof($$n)>0 && print("[{$n}]\n") ) print_r($$n);return ob_get_clean();');
  foreach (array('_GET','_POST','_COOKIE','_SESSION','_ENV','_FILES','_SERVER','_REQUEST','HTTP_POST_FILES','HTTP_POST_VARS','HTTP_SERVER_VARS','HTTP_RAW_POST_DATA','HTTP_GET_VARS','HTTP_COOKIE_VARS','HTTP_ENV_VARS') as $k)echo $gv($k);
  print_r(get_defined_constants());
}

Also check the WordPress Codex page: Editing wp-config.php and Perishable Press’s: Stupid WordPress Tricks

Advanced WordPress wp-config.php Tweaks originally appeared on AskApache.com

The batch file also runs PortaPuttY plink to create forwarded tunnels from various remote servers and accounts, all using key-based encryption. This includes dynamic SOCKS 4/5 tunnels, VPN tun device tunnels, and of course the basic SSH port forwarding tunnels that are so powerful. These tunnels are automatically reconnected if they are disconnected, using simple windows builtin command-line tools. And believe me it was not easy to figure out how to make this all work using plink ( essentially the same as putty minus the gui ), I literally had to use almost all of my Windows kung fu to finally end up with this.

Using MyEnTunnel

Initially I was using the MyEnTunnel program combined with a custom windows batch install script I wrote to handle the tunnels.

The tunnels are very important to making things easy while improving security. It’s not easy to understand at first, but basically it means you can now connect to ANY IP address:port as if you were on that very machine connecting to localhost, like if you pinged yourself!. The result is any traffic you want is now encrypted, and you can set up your servers to only accept connections from localhost, which could save you tons of memory, bandwidth, and security attack vectors to think about. So I configure everything to use these tunnels as proxies, like Mozilla Thunderbird and Chrome, Firefox, Pidgeon, all portable versions and running from my encrypted usb drive.

This means you can walk into my house with that usb key, plug into any computer here, and surf the web/check your emails all across SSH… I know for a fact I wouldn’t be able to snoop that traffic! There is a lot of exciting things going on around here, new servers and all.. Its going to take a couple more posts for me to finish this up, enjoy the article and comment.

Buy a couple USB Mini Drives

The first thing to do, is purchase a USB thumb drive.. My favorite store, TigerDirect.com, has over 104 tiny usb drives for under $24.. I’ve used them since the late 90′s.

I bought some 4GB PNY’s the size of a fingernail at a gas station and they are amazing, way faster than say a dvd drive. Just try to do some research of the differences between the 16GB vs the $4 1GB drives.. You want speed because the whole drive will be encrypted. If you can afford the super excellent and crazy fast ones, hey send me one! Buying cheap means you can buy 3 or 4 so you can always have backups. This device will make you Internationally mobile, untethered from a box, maybe getting some work done at a cafe in Florenze, or at a beach hotel in Miami. Keep dreaming, but that is more possible with a better organized system.

Backup the USB Drive

You only need to know 1 way that works, there are several. The way I do backups is to copy the entire disk image of the usb, that way I can always access it in case of usb key failure, which does happen. Free software like CloneZilla live CD with its crazy cluster computing power, or Self Image, which is free for both linux and Windows. And you could never go wrong with GHOST, one of the first to make mega bucks in the market.. it’s some seriously impressive software but not open-source. Even easier for some is to just set a cron job for dd to pipe the entire drive image to a remote computer using netcat, or sshfs, or curlftpfs, or just simple ssh like below. Once setup (without stupid, bulky, dangerous software), the files on your encrypted usb don’t change often, otherwise I would want to sync a backup to happen automatically every X number of logins or days (test logfile time in bash_profile?)..

SSH Back-ups To Remote Server

Files and data on your drives slow it down tremendously, meaning a web server storing backups locally is slower than one storing them externally.

Notice how much safer this command is by optimizing both the CPU and DISK I/O.. Though it’s much smarter to create a new separate ssh user, one with no shell and a passwordless safer key-based encryption. Then in your /etc/security/limits.conf file or your initscript you can cause that user to have nice -19 and ionice -c2 -n7 priority set all the time automatically, since sshd, compression, and disk writing are this accounts only job. turboslow is an alias defined in a ssh_config file so you don’t have to type the host, port, and settings each time.

#
# much better ways to do this on google!!!!!!!
#
ionice -c2 -n7 nice -n 19 dd if=/dev/sdb2 bs=1k conv=sync,noerror | gzip -c | ssh turboslow "dd of=sdb2.gz bs=1k"

Note that you may decide it would be better to configure the ssh connection to a less CPU intensive algorithm, perhaps even protocol 1 and DES. That’s perfectly alright, but the tradeoff is that the encryption can be broken much quicker, and so you would have to implement a cron job to create new keys on both ends of the tunnel every few hours.. It’s really not a big deal to setup, kind of sweet way to use key-based encryption. Also, important files ( those containing passwords, any database ) are encrypted before transport using private GPG keys, which don’t need to be changed. The other thing to think about too is only letting your main PC send/write on the backup host, so the backup host is only authorized to rx and can never login back to yours.

Hey! the Internet is a dangerous place you better believe it! And it’s only going to get more interesting with cloud computing’s breakthrough’s… More people who know they’re way around… I can always use an extra server, I’d love to expand my network another node without having to pay for it (free cloud computing?), so make sure your servers are locked up strenuously. Not super perfect, just a little unique or creative in your defense to avoid any coming super-worm’s that may be employing vast arsenals of the deadliest attack-engines like metasploit.. Scarry rumors.

Compression Speeds: PBZip2, Rzip, Lzop, Gzip

Probably the fastest is to use rsync over ssh, which is what I’m doing, since the algorithms used by rsync are much faster and safer. Rsync also lets you specify a compression program, so depending on your machine you will want pbzip2 (for multi processors) or rzip which are the 2 fastest I know of, though I have had some reliability issues with rzip for gigabyte transfers. Pbzip2 is amazing, blew me away the first time being 8x faster (8 CPUs) then anything. You can get it and compile a static binary for your thumb drive if want at Parallel BZIP2 (PBZIP2). Heavy code, re: this note by Jeff Gilchrist

NOTE: If you are looking for a parallel BZIP2 that works on cluster machines, you should check out MPIBZIP2 which was designed for a distributed-memory message-passing architecture.

tar cpf "$G" --use-compress-prog=pbzip2 ./

Benchmarking for Performance

Finally a couple tips, you should get an idea what the device can do, format it a few times for linux and test it on windows, and vice versa.. Some drives are too small or too old and can only support fat32 filesystems on winblows, you DO NOT want fat32 because this drive is going to be 100% encrypted and then 100% transparently decrypted as you use it,

# note this is 512MB
dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
512000000 bytes (512 MB) copied, 5.16588 s, 99.1 MB/s

Part II: Encrypted AutoRunning USB Key with TrueCrypt

Now this section anyone can do, it’s so easy on Windows. What I’m going to show you how to do is get setup the right way super-fast. There are many ways to use TrueCrypt, it’s one of the nicest built software programs’s I’ve ever used… Sadly, it is not licensed open-source, and that is often a deal-breaker for security-conscious folks or anti-pirate anarchists. From the very helpful TrueCrypt web site:

• Creates a virtual encrypted disk within a file and mounts it as a real disk.
• Encrypts an entire partition or storage device such as USB flash drive or hard drive.
• Encrypts a partition or drive where Windows is installed (pre-boot authentication).
• Encryption is automatic, real-time (on-the-fly) and transparent.
• Parallelization and pipelining allow data to be read and written as fast as if the drive was not encrypted.
• Provides plausible deniability, in case an adversary forces you to reveal the password: Hidden volume (steganography) and hidden operating system.
• Encryption algorithms: AES-256, Serpent, and Twofish. Mode of operation: XTS.

Further Reading

• Network File Copy using SSH
• Check out the trunk version of PuTTY:~ svn co svn://svn.tartarus.org/sgt/putty

The real fun doesn’t start till all the automation starts, automating all of that from a couple batch files I wrote, one click setup. Kind of like building your own knoppix for when you have to use Windows. To begin this tutorial, setup a truecrypt traveller setup on your usb and also install the portaputty package onto the usb. You do this by creating a 3GB or whatever file on the usb and then mounting that file like you would mount an iso file. I will show the Windows Batch file I use and the tricks with Windows Volume names and how to consistently make it all work. Then we will setup MyEnTunnel with a customized batch file that forces all puttys to use portaputty (sweet hack stolen from sysinternals pagedefrag tool).Stay Tuned!

PortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG originally appeared on AskApache.com

I use this awesome skeleton setup for all my high-paying clients sorry poor people! and also of course on this blog, which I use as a bleeding-edge dev server for my crazy testing. So realize that I’m already past this setup and using it to do cooler stuff. In order for you to use these more advanced ideas, you first need to get up to speed on what I’m doing so you know what I’m talking about. This article tries to help you accomplish that… remains to be seen.

An Optimized Website, The Real Deal

This first article is to give you some ideas and get you thinking and reading before the first article in this series comes out. This series details how to setup, configure, secure, optimize, and manage a website the best possible way I can come up with. It pieces together all the AskApache hacks and tricks and uses methods and ideas discussed all over this blog and all over the net and glues them all together to show you how to have the most optimized, fastest, best website setup I can think of.

Knowing the why and how behind the operation of a Web Server allows us to optimize that operation. For this example we will be creating the website www.askapache.com, which will be running WordPress and php. We will also set up static.askapache.com to serve all of our sites uploads, images, css and javascript files, flash files, etc. with advanced caching and security using Apache Server .htaccess files. So lets get started and take a look at this site structure for a moment.

/home/askapache.com
|-- /home/askapache.com/backups/
|-- /home/askapache.com/public_html/
|-- /home/askapache.com/inc/
|-- /home/askapache.com/logs/
|-- /home/askapache.com/static/
|-- /home/askapache.com/tmp/
|-- /home/askapache.com/.htpasswd-basic
`-- /home/askapache.com/.htpasswd-digest

• /backups/ – For encrypted backups of WordPress database and site files. And any other backups.
• /public_html/ – The document root for www.askapache.com
• /inc/ – Folder to keep your php include files for extra security and easy management.
• /logs/ – Save your php, apache, and other logs here or create symlinks to them.
• /static/ – The document root for static.askapache.com
• /tmp/ – Only need this if your host doesn’t already have a /tmp folder

Strong Security, Top to Bottom

Simply by implementing correct access permissions, file permissions, password protection and segmenting various folders and services we are already ahead of the game. I’ve always taken security extremely seriously, so you can benefit from alot of the simple solutions I’m recommending for a really locked down site.

Indeed, security is a major part of every step of this setup process, as security concerns are what drives a lot of the motivations I have for coming up with this setup in the first place. We will be doing very simple but very effective site security like the following items, which is a short list compared to everything we will be doing.

• Fixing file permissions automatically
• Searching for modified files on the server
• Encrypting your backups
• Get alerted to breakin attempts
• Block tons of bad clients
• Disallowing cgi scripts or any other handlers, just serve files.
• Configuring PHP
• Password Protection for certain areas

Ready for Warfare?

My past work for an Internet Service Provider, followed by 4 years of auditing the security of organizations external/internal networks has given me a fresh perspective on website security, and I think it allows me to see what would really be effective at preventing and killing attacks. In fact just last night I was once again doing some research into some off-the-wall security topics, and I discovered a new defense method that I will be writing about very soon. I believe that this new method, could be quickly adopted and implemented by hosting providers and software developers, which would result in us finally taking the Internet back from all those zombies and robots. This method will be discussed in great detail soon, and will be a core part of this site setups security and optimization.

Built to Bleed Speed

Serve’s files as fast and efficiently as possible using advanced caching, HTTP Protocols, php/server configurations.

Many of the articles and research on this blog is about improving the speed and efficiency of your website. In fact that is why I am helping develop open-source software to block spammers from WordPress blogs… not because I’m bothered by the spam, but because they make the net slow! So lets look at some of the ideas we’ll be implementing.

Many techniques I’ve been using and tweaking for several years, and recently many of them were included in the high-performance websites list. Of course we will be taking a look at this list in practical terms, meaning almost all of it, the caching, compression, etc., will be automated in keeping with our “comfort” goal, which is to say we want to make the Web Developer and Server Admin’s lives as easy and comfy as possible. After all, we do the work right?

1. Reduce HTTP requests – Reducing 304 requests with Cache-Control Headers
2. Use a customized php.ini – Creating and using a custom PHP.ini
3. Add an Expires header – Caching with mod_expires on Apache
4. Gzip components
5. Make CSS and unobtrusive Javascript as external files not inline
6. Reduce DNS lookups – Use Static IP address, use a subdomain with static IP address for static content.
7. Minimize Javascript – Refactor the code, compress with dojo
8. Avoid external redirects – Use internal redirection with mod_rewrite, The correct way to redirect with 301
9. Turn off ETags – Prevent Caching with htaccess
10. Make AJAX cacheable and small

AskApache.com, Fastest Site Ever!

Ok it might not be the #1, but surely the top 10.. ;)

I’m very proud of the performance I am able to achieve on this site. Very proud. I started looking for ways to improve the wp-cache and wp-super-cache WordPress plugins, and came up with hacks for both of them.. but they still didn’t do what I wanted so I started from scratch and wrote my own caching plugin.

With much more advanced caching options and unquestionably higher performance and lower time usage on the machine. I’m hesitant to release it to the public until I get faded on it.. I just really love it.. it has been running my site for several months now and I keep finding ways to improve it.. Stay tuned.

One feature it has is the ability to allow negotiation of a resource between apache and the client. Think almost transparent mutli-lingual blogs, mutliple formats per document (look at the rdf for this page for an example*). But that plugin is the future and this is the present.. so back to it we go.

Pamper the Webmaster with Extreme Comfort

This section alone would make this setup appealing. I have developed all types of techniques and methods to make my life as easy as possible. I could literally DIE right now and this blog would continue to run and operate for years on its own. The general philosophy that I have used to get to where I can goto the beach with my laptop and do all this crazy stuff is the idea of perfection. That may sound a little put-offish, but it basically means I will focus in on one very specific area for improvement or research and just get sick with it. Most of this blogs articles are enlightening examples of this in action. I will take a relatively unknown or unused piece of code or software and experiment with it until I feel I have it down, then I move on to the next item of never-ending research. Mostly I think this is just plain habit from when I was studying security. I’m much better at this then that :)

Apache ErrorDocuments

The 57 HTTP Status Codes and Apache ErrorDocuments article is a prime example. I was simply searching for an authoritative list of HTTP status codes, an issue not many web people find worth their time, and that search led to some of the most useful stuff I’ve found about the Web

This “Comfort” article will include multi-language, intelligent, and optimized error documents for handling any type of HTTP error with class and allow us to stop spammers, save bandwidth, redirect correctly, etc.. You will probably be surprised at all the uses an Apache ErrorDocument can have.. It IS one of the foundations of the HTTP-based Net after all.

Emphasis on Easy Upgrades

The whole setup is geared to make hassle-free WordPress/PHP/application upgrades possible by keeping different types of files in separate places, keeping backups, other misc tricks and since all of these files are in /home/askapache.com, your FTP connection can still access every file easily. Sometimes security and optimizing your server can lead to it being more of a pain to do updates, backups, and general maintenance. This article tries to overwhelm the balance with a trifecta of goals.

Move to a new host? Ok!

Comfort to me also means being able to pack up the whole website and database and move to another web host in under an hour. I can move the whole AskApache site to one of several other hosting providers accounts I have in about 30minutes. If this was a clients site or I was getting paid more, I’d also be focused on round-robin DNS technology, balance-load setups, and just go crazy making it fast.

Staying Online, Improving Uptime

Ever since I started sharing information and software to stop all these resource hogging zombies attacking everything I’ve been attacked several times. Normally I get over 10K exploit attempts or requests per day, which I pretty much block 100%. But a few times they’ve actually tried to DDOS me off the net in a distributed attack. I have implemented several “poor mans” techniques to put up your best effort at surviving, which I did. Basically you want to configure your server to KILL connections just as fast as possible and prevent your server resources from skyrocketing and surpassing your quotas. A skilled attacker could easily shut you down even without the use of a widespread botnet if they are clever, which could be devastating to your small blog or site if it goes down at a crucial instant.

Organization with Templates and Systems

I used to work with a guy who did alot of the coldfusion programming for us, and I used to cringe every time I was called in to upgrade a site or do a re-design. Files and folders EVERYWHERE! Literally images in every folder, multiple index.html, index1.html, index-old.html, and on and on it went.. It would take me hours just to reverse-engineer the site enough so I could modify files on it without having some unkown consequence happen.

Do You Have a Cluttered Desktop?

Everyone has this problem, what I do all the time is just grab everything on my desktop and put it in a folder named with the date. Then the process repeats itself and invariably a few months later I’m looking at a cluttered screen again.

This absolutely is the worst thing that can happen to a website, worst for security, comfort for webmaster, and speed. So this setup addresses that issue completely heads on. With all the different pages, tools, and resources available on this blog, I can almost promise you that my site has less files than yours. No small feat to be sure, but worth every second I spent researching how to do it now that its on and popping.

What’s a Website really?

All hosts are different, but any host worth their salt is running some kind of BSD/Linux operating system, and that is good news because those operating systems all use very similarly excellent file/folder structures with huge organization systems. If your web hosting provider is running on a Windows based operating system or other locked/proprietary software than this article is not for you and I would recommend switching hosts to a BSD/Linux open-source operating system.

Listening for Requests with Web Hosting and DNS

First you set your website up so it can start serving.

1. You buy your domain name, which just gives you the right to use it.
2. You pay your webhost for an account on their machine running a Server connected to the Net via a fast connection link.
3. You pay a DNS provider to redirect requests for your domain name to be sent to your webhosts machine running the server.

Sub-Domain for Serving Assets

This is a very cool method I’ve been using more and more frequently because it makes updates, upgrades, and changes so much easier to manage. And segmenting various parts of the site is smart security, and even smarter in the way of speeding up a website and keeping your servers running mean and lean.

Full Site Structure Expanded

/home/askapache.com
|-- /home/askapache.com/backups/
|-- /home/askapache.com/public_html/
|   |-- /home/askapache.com/public_html/about/
|   |-- /home/askapache.com/public_html/admin/
|   |-- /home/askapache.com/public_html/cgi-bin/
|   |-- /home/askapache.com/public_html/.htaccess
|   |-- /home/askapache.com/public_html/index.php
|   `-- /home/askapache.com/public_html/robots.txt
|-- /home/askapache.com/inc/
|   |-- /home/askapache.com/inc/config.inc.php
|   `-- /home/askapache.com/inc/settings.inc.php
|-- /home/askapache.com/logs/
|   |-- /home/askapache.com/logs/access.log
|   |-- /home/askapache.com/logs/error.log
|   |-- /home/askapache.com/logs/logins.log
|   |-- /home/askapache.com/logs/modsec_audit.log
|   |-- /home/askapache.com/logs/modsec_debug.log
|   `-- /home/askapache.com/logs/php_error.log
|-- /home/askapache.com/static/
|   |-- /home/askapache.com/static/css/
|   |-- /home/askapache.com/static/flv/
|   |-- /home/askapache.com/static/img/
|   |-- /home/askapache.com/static/js/
|   |-- /home/askapache.com/static/mp3/
|   |-- /home/askapache.com/static/pdf/
|   |-- /home/askapache.com/static/swf/
|   |-- /home/askapache.com/static/.htaccess
|   |-- /home/askapache.com/static/index.html
|   `-- /home/askapache.com/static/robots.txt
|-- /home/askapache.com/tmp/
|-- /home/askapache.com/.htpasswd-basic
`-- /home/askapache.com/.htpasswd-digest

Full Expanded Structure

/home/askapache.com
|-- /home/askapache.com/backups/
|-- /home/askapache.com/public_html/
|   |-- /home/askapache.com/public_html/about/
|   |   `-- /home/askapache.com/public_html/about/index.html
|   |-- /home/askapache.com/public_html/admin/
|   |   |-- /home/askapache.com/public_html/admin/.htaccess
|   |   `-- /home/askapache.com/public_html/admin/index.html
|   |-- /home/askapache.com/public_html/cgi-bin/
|   |   |-- /home/askapache.com/public_html/cgi-bin/bin/
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/bin/.htaccess
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/bin/php.cgi*
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/bin/php.ini
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/bin/php4.cgi*
|   |   |   `-- /home/askapache.com/public_html/cgi-bin/bin/php5.cgi*
|   |   |-- /home/askapache.com/public_html/cgi-bin/private/
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/private/.htaccess
|   |   |   |-- /home/askapache.com/public_html/cgi-bin/private/debug.php
|   |   |   `-- /home/askapache.com/public_html/cgi-bin/private/stats.php
|   |   |-- /home/askapache.com/public_html/cgi-bin/.htaccess
|   |   |-- /home/askapache.com/public_html/cgi-bin/login.php
|   |   |-- /home/askapache.com/public_html/cgi-bin/printenv.cgi*
 
|   |   `-- /home/askapache.com/public_html/cgi-bin/redir.cgi*
|   |-- /home/askapache.com/public_html/.htaccess
|   |-- /home/askapache.com/public_html/index.php
|   `-- /home/askapache.com/public_html/robots.txt
|-- /home/askapache.com/inc/
|   |-- /home/askapache.com/inc/config.php
|   `-- /home/askapache.com/inc/functions.php
|-- /home/askapache.com/logs/
|   |-- /home/askapache.com/logs/access.log
|   |-- /home/askapache.com/logs/error.log
|   |-- /home/askapache.com/logs/logins.log
|   |-- /home/askapache.com/logs/modsec_audit.log
|   |-- /home/askapache.com/logs/modsec_debug.log
|   `-- /home/askapache.com/logs/php_error.log
|-- /home/askapache.com/static/
|   |-- /home/askapache.com/static/css/
|   |   `-- /home/askapache.com/static/css/apache.css
|   |-- /home/askapache.com/static/flv/
|   |   `-- /home/askapache.com/static/flv/apache.flv
|   |-- /home/askapache.com/static/img/
|   |   |-- /home/askapache.com/static/img/apache.gif
|   |   |-- /home/askapache.com/static/img/apache.jpg
|   |   `-- /home/askapache.com/static/img/apache.png
|   |-- /home/askapache.com/static/js/
|   |   `-- /home/askapache.com/static/js/apache.js
|   |-- /home/askapache.com/static/mp3/
|   |   `-- /home/askapache.com/static/mp3/apache.mp3
|   |-- /home/askapache.com/static/pdf/
|   |   `-- /home/askapache.com/static/pdf/apache.pdf
|   |-- /home/askapache.com/static/swf/
|   |   `-- /home/askapache.com/static/swf/apache.swf
|   |-- /home/askapache.com/static/.htaccess
|   |-- /home/askapache.com/static/index.html
|   `-- /home/askapache.com/static/robots.txt
|-- /home/askapache.com/tmp/
|-- /home/askapache.com/.htpasswd-basic
`-- /home/askapache.com/.htpasswd-digest

The buzz about apache and open-source is very real, apache is becoming more of a discussed topic as people realize the power and importance of Doing it and Doing it and Doing it well. – Movable Type Apache Installs made easy, Checking out Apache Web logs, Securing WordPress with .htaccess, WordPress Permalinks and .htaccess, New search and replace module for apache!, creating an .htaccess template, .htaccess allow directive

Check back in a week for the first article, or better yet subscribe to my rss feed or use the comment form below to get notified.

Optimize a Website for Speed, Security, and Easy Management originally appeared on AskApache.com

Top 3 WordPress Speed Plugins

1. DB Cache Reloaded
2. WP Super Cache
3. AskApache Crazy Cache

WP Caching Overview

Each request to your blog has to fire up the php interpreter and query the mysql database to create the output that you see in your browser. Using a plugin like WP Super Cache simply saves the output of that request as a static HTML file and serves that to a request instead of using php and mysql every time. DB Cache Reloaded takes this a step further by optimizing the mysql queries. Finally, AskApache Crazy Cache is used to keep a cache fully primed and ready.

Why Caching?

If you have a private server, or you want to keep your MEMORY, BANDWIDTH, and CPU usage down for your server, these plugins will be dramatic. If you have a site that is updated maybe once a month and gets a very small amount of traffic, then the AskApache Crazy Cache would be redundant. That plugin is geared for the heaviest traffic sites.

Request and Response

Most HTTP communication is initiated by a user agent (browser) and consists of a request to a resource on some origin server. In the simplest case, this may be accomplished via a single connection (v) between the user agent (UA) and the origin server (O).

             request chain ------------------------>
          UA -------------------v------------------- O
             <----------------------- response chain

If a browser requests a WP-driven page, the server generates the response (the outputted html) by loading a php interpreter or module to read the WP php files and load all the settings. WP stores the settings in a mysql database and has to query the mysql database for all the data (like the content of your post that becomes html). Finally php sends the output through the server back to the browser. This is the norm for most PHP applications. Every time an interpreter is loaded additional CPU and Memory are used. And perhaps even more troublesome for shared-hosting using a virtual or network filesystem, each load causes many hard-disk accesses, additional processes, etc.

Cache PHP

By saving the php-generated output of a page to a static html file, your server can entirely skip loading a php interpreter or other process. Originally servers were needed and created to essentially open a static file on disk and send that file back to the requesting user. So servers are specialized for this as it’s their core function.

WP-Super cache does this for you by saving the output into a static html file, and by instructing the server to skip loading php. It basically only uses php for creating the cached version, in the same way that you can save this webpage as an html file, WP-Super cache saves all the WP-blog permalinks to static files.

DB-Cache Reloaded does something entirely different, it saves the mysql queries that are made to the WP-database, as well as the mysql results to static files, and then through php serves those cached-files instead of re-querying the mysql database. Most mysql databases are stored on separate servers, and although many are on the same local network there is a limit to how many queries, and how many connections can take place. But mysql is maybe the fastest thing I’ve seen, so your bottlenecks almost never happen there (if configured correctly).

So DB-Cache Reloaded basically makes WP-Super Cache work alot faster when generating the cache files, and DB-Cache Reloaded helps in a number of areas un-related to WP-Super Cache, like in the admin panel. And DB-Cache without WP-Super-Cache is a joke because it still uses the application-level and php for everything. Gotta use both (or just WPSC).

AskApache Crazy Cache is a plugin I wrote to do one thing very well, it runs at intervals via the WP-cron and forces WP-Super-Cache to create a static cache file for all the posts, pages, etc. on your site. Without this WPSC likes to do dumb things like try to manage it’s own cache with stale files and expired files, which equals a lot more php interpreters getting loaded instead of cached static files. For sites with more than a visit/page/10minutes this plugin keeps a full primed cache built by WPSC.

Compression and WP-Super Cache

Enabling compression in WP-Super Cache is almost always a great idea. I’ve never had a problem other than some php compat issues with not-updated php installations. This option basically lets WP-Super Cache compress the generated output of a page and save that to the static file. Normally Apache, Lighttpd, Nginx, etc. open the static file and compress it before it is sent to the browser, then the browser automatically decompresses it to view. This happens so fast because it is run by the server.

This lets WPSC instruct your server to send the compressed version to all browsers that accept compression, and send the uncompressed static file to any other browsers. So this is helpful because it eliminates your server having to do any transparent compressing, it can instead just focus on what it does best, serving static files.

PHP is an application so it requires memory, hard-drive access, and CPU time. Check out the protocol hierarchy:

       +------+ +-----+ +-----+       +-----+
       |Telnet| | FTP | |Voice|  ...  |     |  Application Level
       +------+ +-----+ +-----+       +-----+
             |   |         |             |
            +-----+     +-----+       +-----+
            | TCP |     | RTP |  ...  |     |  Host Level
            +-----+     +-----+       +-----+
               |           |             |
            +-------------------------------+
            |    Internet Protocol & ICMP   |  Gateway Level
            +-------------------------------+
                           |
              +---------------------------+
              |   Local Network Protocol  |    Network Level
              +---------------------------+       

DB Cache Reloaded

DB Cache Reloaded Plugin Page – News

This plugin caches every database query with given lifetime. It is much faster than other html caching plugins and uses less disk space for caching.

I think you’ve heard of WP-Cache or WP Super Cache, they are both top plugins for WordPress, which make your site faster and responsive. Forget about them – with DB Cache Reloaded your site will work much faster and will use less disk space for cached files. Your visitors will always get actual information in sidebars and server CPU loads will be as low as possible.

Why is DB Cache Reloaded better than WP Super Cache?

This plugin is based on a fundamentally different principle of caching queries to database instead of full pages, which optimises WordPress from the very beginning and uses less disk space for cache files because it saves only useful information. It saves information separately and also caches hidden requests to database.

WP Super Cache

WP Super Cache Plugin Page – News

This plugin generates static html files from your dynamic WordPress blog. After a html file is generated your webserver will serve that file instead of processing the comparatively heavier and more expensive WordPress PHP scripts.

The static html files will be served to the vast majority of your users, but because a user’s details are displayed in the comment form after they leave a comment those requests are handled by PHP. Static files are served to:

· Users who are not logged in.
· Users who have not left a comment on your blog.
· Or users who have not viewed a password protected post.

99% of your visitors will be served static html files. Those users who don’t see the static files will still benefit because they will see regular WP-Cache cached files and your server won’t be as busy as before. This plugin will help your server cope with a front page appearance on digg.com or other social networking site.

Why is WP-Super-Cache better than WP-Cache?

This plugin is based on the excellent WP-Cache plugin and therefore brings all the benefits of that plugin to WordPress. On top of that it creates copies of every page that is accessed on a blog in a form that is quickly served by the web server. It’s almost as quick as if the you had saved a page in your browser and uploaded it to replace your homepage.

Will the Super Cache compression slow down my server?

No, it will do the opposite in fact. Super Cache files are compressed and stored that way so the heavy compression is done only once. These files are generally much smaller and are sent to a visitor’s browser much more quickly than uncompressed html. As a result, your server spends less time talking over the network which saves CPU time and bandwidth, and can also serve the next request much more quickly.

AskApache Crazy Cache

AskApache Crazy Cache Plugin Page – News

This sweet little plugin does one thing very well. It caches all the posts on your entire blog at the same time, if you are using WP-Cache, WP-Super-Cache, or Hyper-Cache.

                              +---------+ ---------\      active OPEN
                              |  CLOSED |            \    -----------
                              +---------+<---------\   \   create TCB
                                |     ^              \   \  snd SYN
                   passive OPEN |     |   CLOSE        \   \
                   ------------ |     | ----------       \   \
                    create TCB  |     | delete TCB         \   \
                                V     |                      \   \
                              +---------+            CLOSE    |    \
                              |  LISTEN |          ---------- |     |
                              +---------+          delete TCB |     |
                   rcv SYN      |     |     SEND              |     |
                  -----------   |     |    -------            |     V
 +---------+      snd SYN,ACK  /       \   snd SYN          +---------+
 |         |<-----------------           ------------------>|         |
 |   SYN   |                    rcv SYN                     |   SYN   |
 |   RCVD  |<-----------------------------------------------|   SENT  |
 |         |                    snd ACK                     |         |
 |         |------------------           -------------------|         |
 +---------+   rcv ACK of SYN  \       /  rcv SYN,ACK       +---------+
   |           --------------   |     |   -----------
   |                  x         |     |     snd ACK
   |                            V     V
   |  CLOSE                   +---------+
   | -------                  |  ESTAB  |
   | snd FIN                  +---------+
   |                   CLOSE    |     |    rcv FIN
   V                  -------   |     |    -------
 +---------+          snd FIN  /       \   snd ACK          +---------+
 |  FIN    |<-----------------           ------------------>|  CLOSE  |
 | WAIT-1  |------------------                              |   WAIT  |
 +---------+          rcv FIN  \                            +---------+
   | rcv ACK of FIN   -------   |                            CLOSE  |
   | --------------   snd ACK   |                           ------- |
   V        x                   V                           snd FIN V
 +---------+                  +---------+                   +---------+
 |FINWAIT-2|                  | CLOSING |                   | LAST-ACK|
 +---------+                  +---------+                   +---------+
   |                rcv ACK of FIN |                 rcv ACK of FIN |
   |  rcv FIN       -------------- |    Timeout=2MSL -------------- |
   |  -------              x       V    ------------        x       V
    \ snd ACK                 +---------+delete TCB         +---------+
     ------------------------>|TIME WAIT|------------------>| CLOSED  |
                              +---------+                   +---------+
 
                      TCP Connection State Diagram

Vetted – Top 3 WordPress Speed Plugins originally appeared on AskApache.com

My Setup

As this screenshot shows, I only use a handful of add-ons at a time. These buttons allow me to clear the DNS/Cookies/Cache for whichever site I’m on when I click it. Very very helpful for me as a web designer. The 4th button there is just a restart button. Other than those, Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.

I like the idea of the last.fm but it’s not as powerful as the site, which is awesome. Lately I’ve been listening to Kings of Leon Radio…

Slow Firefox

The more add-ons you have, disabled or not, the slower Firefox is. (unless you are running your profile folder in TMPFS). Also, bookmarks and settings like that have a performance hit. I have been using Firefox since it launched way back when, and I have always kept my bookmarks when moving to new machines and new installations.. So with over 5 thousand bookmarks I finally did some debugging and discovered that was a huge cause of Firefox acting slow. Now I am trying to migrate them all over to Google’s Gmarks, which knowing Google will be awesome eventually.

Multiple Firefox Installations, Sorta

The solution to all these problems is to use separate Firefox profiles which use separate folders to store your profile-specific extensions and settings. So I have profiles with upwards of 40 Addons installed and enabled, and another profile that is built for speed… It’s very very slow to be running Firebug and have multiple tabs open.. You can use the profilemanager to load them specific profiles from the command line. I personally use separate icons on my Windows Quick Launch that I just modified the shortcut pointing to firefox to also have the profile commandline. Note also that you can have multiple profiles open and running simultaneously.. This lets you do some crazy networking and other random hacks like having many Firefox instances running each of which is configured to use a separate SOCKS Proxy or network interface, so you can really open up those pipes for some intense txrx.

You can do some very powerful things with Firefox that most people are unaware of, if you are interested start with these:

• Setting up an extension development environment
• Firefox Command Line Options

About This Collection

Web Development Add-ons for Advanced Web Developers. I personally use these to work on Sites, Servers, WordPress, Javascript, PHP, CSS, XHTML, validation, page-loading, SEO, optimizing, and much more. Add this collection.

Created by:AskApache

Updated:September 22, 2009

Share this Collection

• Digg this!
• Post to Facebook
• Add to Delicious
• Post to MySpace
• Share on FriendFeed
• Post to Twitter

Firefox Add-ons for Web Developers originally appeared on AskApache.com

Keep in mind that this blog and my research is only a hobby of mine, my job is primarily marketing and sales, so I’m not some licensed expert or anything, or even an unlicensed expert! haha. But it does bother me that those who are tech-savvy enough to run web-hosting companies are happily ripping people off. So this article details the main tools that are used to speed up and optimize your machine by delegating levels of priority to specific processes. Future articles will use these tools alot, so this is meant as an intro.

CPU and Disk I/O

As most of you are aware, there are 2 variables that determine any computer or programs speed. CPU and Disk I/O. CPU determines how fast you can process data, crunch numbers, etc. while disk I/O determines how fast your disks can read and write data to the hard-drive. Wouldn’t it be great if you could easily configure your server to give your httpd, php, and other processes both greater CPU processing and disk IO than your non-important processes like backup scripts, ftp daemons, etc.? We are talking about Linux in this article, so of course YES not only can you do that, you should!

RAM

RAM is like a hard-drive in that data is stored on it, and read/written to it. The difference is that RAM is somewhere around 30x faster than disk I/O, but the cost of that incredible speed is that the data stored on it is only temporary in the sense that it won’t be stored permanently, it is completely erased when your machine is rebooted. RAM is also expensive, and there is a limit to how much a server or machine can have due to hardware limits.

SWAP

SWAP takes off when you run out of RAM but you still want certain data to be read/write quickly. Basically when you start running out of RAM your machine starts supplementing RAM with SWAP storage. SWAP is usually a partition on a second hard-drive disk. There is an upper limit on how much I/O can occur on a disk at one time, and the more I/O takes place, the slower all I/O becomes, so SWAP works well on a separate hard-drive as it will have much faster I/O. On Windows they opted to copy the SWAP mechanism but instead use a file named pagefile.sys, and that is just one reason people in the know do not care for Windows.

CPU

So lets do this, think of your CPU (your processor) as having an amount of 100% processing available when not being used, 0% when its maxed out. CPU’s handle multiple processing tasks simultaneously, so what we will discuss in this article is how to specify HOW MUCH of that processing amount each of your programs (heretofore “processes”) are able to use. Yes, very very cool.

That is correct, you can easily configure your server to provide more of the available processing time to certain programs over others, like you can configure apache and php to utilize 50% of your CPU processing time by themselves, so that all other processes (proftpd, sshd, rsync, etc.) combined can only utilize 50%. The terminology is we can give certain specific processes (like php.cgi, httpd, fast-cgi.cgi) a specific priority, where -19 is the most priority, and +19 is the least amount of priority, or CPU processing time. I know it seems backwards..

The Tools

If you run Windows, you are in the right place… because the following advice will save your life: GET LINUX! Ok, now that that is out of the way, the following are the tools dicussed on this page. All of them are free, open-source, and wonderful. The basic idea of these tools is to control how much CPU is devoted to each process, and also how much Disk IO/Disk traffic is given to each process.

nice

run a program with modified scheduling priority

renice

alter priority of running processes

ionice

set or retrieve the I/O priority for a given pid or execute a new task with a given I/O priority.

iostat

Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.

ulimit

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control.

chrt

set or retrieve real-time scheduling parameters for a given pid or execute a new task under given scheduling parameters.

taskset

set or retrieve task CPU affinity for a given pid or execute a new task under a given affinity mask.

Part 1: Process Processes Faster

Ok so lets tackle figuring out how to give your response-intensive processes (like apache, php, ruby, perl, java) meaning a request to your server/machine requires a response. For instance, when you requested this page that you are reading at this very second, several things on my server had to happen for you to be able to read this.

First your computer sends out a request to see what server the www.askapache.com domain name is. DNS servers respond with my server IP, so for servers dedicated as nameservers, optimizing the DNS processes like bind would speed that up. Now that your computer knows how to reach my server it sends an HTTP GET request for this url. This request is received by the httpd process that is apache, and apache determines this url should be handled by my custom compiled php5.3.0 binary, because this page is WordPress generated. So the php binary loads up the WordPress /index.php file, which chain-loads several other php files, including wp-config.php containing my MySql database settings. Now php connects to my MySql Server to fetch this articles content, comments, title, tags, etc. and then generates the HTML and hands that back to Apache.

Finally, Apache generates a HTTP RESPONSE and sends the RESPONSE and CONTENT back to your Browser, which then in turn renders the page for your eyes with the necessary javascript, images, css, and other files included in the HTML response.

Too much Processing

Now you see why I’ve opted to write my own caching plugin that takes the php and mysql processes OUT of that equation. Both the php binary and the mysql instance consume CPU processing, and disk IO, to load all their library files, make various network requests and sockets, check permissions, and on and on. And that’s completely ok, the thing is, unless you configure these processes (Apache, PHP, MySQL) they will use the same amount of CPU processing that other processes use, other processes that have very little to do with you reading this sentence. Processes to run my mail server, my FTP server, my SSH server, my cronjobs, cleanup scripts, atd daemon, etc.. and they will get the same amount of CPU!

Another even simpler example is what got me to look into this myself. I wrote a shell script that created hourly, daily, weekly, and monthly backups for all of my websites and sql databases, and set it up to run by cronjob at those set intervals. Eventually I noticed my sites were slower, my php even slower, and sometimes I even saw 503 errors that my host throws up when my server is overloaded. The research that I pursued to prevent that from happening has been hugely eye-opening. What does a backup script do? Mine just created tar archives of all the files in my web root, then gzipped the tar archive saving to a backup server using scp (a file transfer using ssh). This resulted in the following huge problems that seem to have nothing to do with a faster server and speedier website, but they have everything to with it.

1. CPU Bottleneck #1 – tar and gzip use compression algorithms at a low level to create a compressed version, and all that compressing uses a whole lot of crunching – CPU processing
2. DISK IO Bottleneck – Tarring the whole web root directory was creating a ton of disk io, and remember the more disk io that is going on, the less is available for everything else.
3. CPU Bottleneck #2 – Using scp to send my backups was security-smart, but these huge archive files had to be encrypted and sent over the net.

Breaking Bottles

I apologize for being a little long-winded there, but I think it’s important to make sure everyone understands those basic concepts, which are foreign to most people. Once you understand what is causing the bottlenecks, then you can understand the solutions, which actually are incredibly simple and even a novice linux user can easily do. Besides, the net gets a little bit faster every time someone implements this.

nice

Nice allows you to run a program with modified scheduling priority which specifies how much CPU is devoted to a particular process. Run COMMAND with an adjusted niceness, which affects process scheduling. With no COMMAND, print the current niceness.

Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable). -n, --adjustment=N – add integer N to the niceness (default 10). nice +19 tasks get a HZ-independent 1.5%. Running a nice +10 and a nice +11 task means the first will get 55% of the CPU, the other 45%.

nice usage

nice [OPTION] [COMMAND [ARG]...]
 
-n, --adjustment=ADJUST   increment priority by ADJUST first

Examples of nice

Using nice to download a file

nice -n 17 curl -q -v -A 'Mozilla/5.0' -L -O http://wordpress.org/latest.zip

Unzipping a file with nice

nice -n 17 unzip latest.zip

Nice way to build from source

nice -n 2 ./configure
nice -n 2 make
nice -n 2 make install

It is sometimes useful to run non-interactive programs with reduced priority.

$ nice factor `echo '2^9 - 1'|bc`
511: 7 73

Since nice prints the current priority, we can invoke it through itself to demonstrate how it works: The default behavior is to reduce priority by 10.

 $ nice nice
10
$ nice -n 10 nice
10

The ADJUSTMENT is relative to the current priority. The first nice invocation runs the second one at priority 10, and it in turn runs the final one at a priority lowered by 3 more.

$ nice nice -n 3 nice
13

Specifying a priority larger than 19 is the same as specifying 19.

$ nice -n 30 nice
19

Only a privileged user may run a process with higher priority.

$ nice -n -1 nice
nice: cannot set priority: Permission denied
$ sudo nice -n -1 nice
-1

The new scheduler in v2.6.23 addresses all three types of complaints:

To address the first complaint (of nice levels being not “punchy” enough), the scheduler was decoupled from ‘time slice’ and HZ concepts (and granularity was made a separate concept from nice levels) and thus it was possible to implement better and more consistent nice +19 support: with the new scheduler nice +19 tasks get a HZ-independent 1.5%, instead of the variable 3%-5%-9% range they got in the old scheduler.

To address the second complaint (of nice levels not being consistent), the new scheduler makes nice(1) have the same CPU utilization effect on tasks, regardless of their absolute nice levels. So on the new scheduler, running a nice +10 and a nice 11 task has the same CPU utilization “split” between them as running a nice -5 and a nice -4 task. (one will get 55% of the CPU, the other 45%.) That is why nice levels were changed to be “multiplicative” (or exponential) – that way it does not matter which nice level you start out from, the ‘relative result’ will always be the same.

The third complaint (of negative nice levels not being “punchy” enough and forcing audio apps to run under the more dangerous SCHED_FIFO scheduling policy) is addressed by the new scheduler almost automatically: stronger negative nice levels are an automatic side-effect of the recalibrated dynamic range of nice levels.

renice

Renice is similar to the nice command, but it lets you modify the nice of a currently running process. This is nice for shell scripts where you can add this to the top of the script to nicify the whole script to 19.

renice usage

renice priority [ [ -p ] pids ] [ [ -g ] pgrps ] [ [ -u ] users ]
 
-g      Force who parameters to be interpreted as process group ID's.
-u      Force the who parameters to be interpreted as user names.
-p      Resets the who interpretation to be (the default) process ID's.

Examples of renice

From the shell, changes the priority of the shell and all children to 19. From a shell script, does the same but only for the script and its children.

renice 19 -p $$

This runs renice without any output

renice 19 -p $$ &>/dev/null

10 gets more CPU than 19

renice 10 -p $$

change the priority of process ID’s 987 and 32, and all processes owned by users daemon and root.

renice +1 987 -u daemon root -p 32

Part 2: Optimizing Disk I/O

Linux Scheduling Policies

The scheduler is the kernel component that decides which runnable process will be executed by the CPU next. Each process has an associated scheduling policy and a static scheduling priority, sched_priority

Processes scheduled under one of the real-time policies (SCHED_FIFO, SCHED_RR) have a sched_priority value in the range 1 (low) to 99 (high). (As the numbers imply, real-time processes always have higher priority than normal processes.) The following “real-time” policies are also supported, for special time-critical applications that need precise control over the way in which runnable processes are selected for execution:

Currently, Linux supports the following “normal” (i.e., non-real-time) scheduling policies:

SCHED_OTHER: Default Linux time-sharing scheduling

The standard round-robin time-sharing policy

SCHED_BATCH: Scheduling batch processes

This policy is useful for workloads that are non-interactive, but do not want to lower their nice value, and for workloads that want a deterministic scheduling policy without interactivity causing extra preemptions (between the workload’s tasks).

SCHED_IDLE: Scheduling very low priority jobs

This policy is intended for running jobs at extremely low priority (lower even than a +19 nice value with the SCHED_OTHER or SCHED_BATCH policies)

SCHED_FIFO: First In-First Out scheduling

A first-in, first-out policy

SCHED_RR: Round Robin scheduling

A round-robin policy.

Scheduling Classes

IOPRIO_CLASS_RT

This is the realtime io class. The RT scheduling class is given first access to the disk, regardless of what else is going on in the system. Thus the RT class needs to be used with some care, as it can starve other processes. As with the best effort class, 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window. This scheduling class is given higher priority than any other in the system, processes from this class are given first access to the disk every time. Thus it needs to be used with some care, one io RT process can starve the entire system. Within the RT class, there are 8 levels of class data that determine exactly how much time this process needs the disk for on each service. In the future this might change to be more directly mappable to performance, by passing in a wanted data rate instead.

IOPRIO_CLASS_BE

This is the best-effort scheduling class, which is the default for any process that hasn’t set a specific io priority. This is the default scheduling class for any process that hasn’t asked for a specific io priority. Programs inherit the CPU nice setting for io priorities. This class takes a priority argument from 0-7, with lower number being higher priority. Programs running at the same best effort priority are served in a round-robin fashion. The class data determines how much io bandwidth the process will get, it’s directly mappable to the cpu nice levels just more coarsely implemented. 0 is the highest BE prio level, 7 is the lowest. The mapping between cpu nice level and io nice level is determined as: io_nice = (cpu_nice + 20) / 5.

IOPRIO_CLASS_IDLE

This is the idle scheduling class, processes running at this level only get io time when no one else needs the disk. A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero. This scheduling class does not take a priority argument. The idle class has no class data, since it doesn’t really apply here.

ionice

ionice – get/set program io scheduling class and priority. This program sets the io scheduling class and priority for a program. Since v3 (aka CFQ Time Sliced) CFQ implements I/O nice levels similar to those of CPU scheduling. These nice levels are grouped in three scheduling classes each one containing one or more priority levels:

ionice usage

If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

ionice [-c] [-n] [-p] [COMMAND [ARG...]]

• -c – The scheduling class. 1 for real time, 2 for best-effort, 3 for idle.
• -n – The scheduling class data. This defines the class data, if the class accepts an argument. For real time and best-effort, 0-7 is valid data.
• -p – Pass in a process pid to change an already running process. If this argument is not given, ionice will run the listed program with the given parameters.

ionice Examples

Sets process with PID 89 as an idle io process.

ionice -c3 -p89

Runs ‘bash’ as a best-effort program with highest priority.

ionice -c2 -n0 bash

Returns the class and priority of the process with PID 89

ionice -p89

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c3 -p16324

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

$ sudo date
$ sudo updatedb &
[1] 16324
$ sudo ionice -c2 -n7 -p16324

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automatted script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

$ ps -C updatedb
PID TTY TIME CMD
4234 ? 00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

iostat

iostat Usage

iostat [ -c ] [ -d ] [ -N ] [ -n ] [ -h ] [ -k | -m ] [ -t ] [ -V ] [ -x ] [ -z ] [ <device> [...] | ALL ] [ -p [ <device> [,...] | ALL ] ] [ <interval> [ <count> ] ]

-c     The -c option is exclusive of the -d option and displays only the CPU usage report.
-d     The -d option is exclusive of the -c option and displays only the device utilization report.
-k     Display statistics in kilobytes per second instead of blocks per second.  Data displayed are valid only with kernels 2.4 and newer.
-m     Display statistics in megabytes per second instead of blocks or kilobytes per second.  Data displayed are valid only with kernels 2.4 and newer.
-n     Displays the NFS-directory statistic.  Data displayed are valid only with kernels 2.6.17 and newer.  This option is exclusive ot the -x option.
-h     Display the NFS report more human readable.
-p [ { device | ALL } ]   The  -p  option  is  exclusive  of  the -x option and displays statistics for block devices and all their partitions that are used by the system.
-t     Print the time for each report displayed.
-x     Display extended statistics.  

iostat Examples

iostat -p ALL 2 1000
avg-cpu:  %user   %nice    %sys %iowait   %idle
            8.34    0.08    1.26    2.27   88.05

Display a single history since boot report for all CPU and Devices.

$ iostat

Display a continuous device report at two second intervals.

$ iostat -d 2

Display six reports at two second intervals for all devices.

$ iostat -d 2 6

Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -x hda hdb 2 6

Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -p sda 2 6

Schedule Utils

These are the Linux scheduler utilities – schedutils for short. These programs take advantage of the scheduler family of syscalls that Linux implements across various kernels. These system calls implement interfaces for scheduler-related parameters such as CPU affinity and real-time attributes. The standard UNIX utilities do not provide support for these interfaces — thus this package.

The programs that are included in this package are chrt and taskset. Together with nice and renice (not included), they allow full control of process scheduling parameters. Suggestions for related utilities are welcome, although it is believed (barring new interfaces) that all scheduling interfaces are covered.

I’ve found that quite a few servers do not have this package installed, indicating to you that they might not know what they are doing. Here is how you can install this incredible package, for non-root users. Root users know how to do this, or they shouldn’t be root. Download and install in 1 line provided you have curl. Or just use the following commands.

mkdir -pv $HOME/{dist,source,bin,share/man/man1} && cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz && cd ~/source && tar -xvzf ~/dist/sch*z && cd sch* && sed -i -e 's,= /usr/local,=${HOME},g' Makefile && make && make install && make installdoc

mkdir -pv $HOME/{dist,source,bin,share/man/man1}
cd ~/dist && curl -O http://ftp.de.debian.org/debian/pool/main/s/schedutils/schedutils_1.5.0.orig.tar.gz
cd ~/source && tar -xvzf ~/dist/schedutils_1.5.0.orig.tar.gz
cd ~/source/schedutils-1.5.0 && sed -i -e 's,= /usr/local,=${HOME},g' Makefile
make || make -d && make install || make install -d && make installdoc || make installdoc -d

taskset

Taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that “bonds” a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given system but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that correspond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. A user must possess CAP_SYS_NICE to change the CPU affinity of a process. Any user can retrieve the affinity mask.

taskset Usage

taskset [options] [mask | cpu-list] [pid | cmd [args...]]
 
-p, --pid            operate on existing given pid
-c, --cpu-list     display and specify cpus in list format

taskset-examples

The default behavior is to run a new command:

$ taskset 03 sshd -b 1024

You can retrieve the mask of an existing task or set it:

$ taskset -p 700
$ taskset -p 03 700

List format uses a comma-separated list instead of a mask:

$ taskset -pc 0,3,7-11 700

chrt

chrt sets or retrieves the real-time scheduling attributes of an existing PID or runs COMMAND with the given attributes. Both policy (one of SCHED_FIFO, SCHED_RR, or SCHED_OTHER) and priority can be set and retrieved. A user must possess CAP_SYS_NICE to change the scheduling attributes of a process. Any user can retrieve the scheduling information.

chrt Usage

chrt [options] [prio] [pid | cmd [args...]]
 
-p, --pid operate on an existing PID and do not launch a new task
-f, --fifo set scheduling policy to SCHED_FIFO
-m, --max show minimum and maximum valid priorities, then exit
-o, --other set policy scheduling policy to SCHED_OTHER
-r, --rr set scheduling policy to SCHED_RR (the default)

chrt Examples

The default behavior is to run a new command: chrt [prio] -- [command] [arguments]

You can also retrieve the real-time attributes of an existing task:

chrt -p [pid]

Or set them:

chrt -p [prio] [pid]

ulimit – get and set user limits

Ulimit provides control over the resources available to processes started by the shell, on systems that allow such control. One can set the resource limits of the shell using the built-in ulimit command. The shell’s resource limits are inherited by the processes that it creates to execute commands.

ulimit Usage

ulimit [-SHacdfilmnpqstuvx] [limit]

-S

use the `soft’ resource limit

-H

use the `hard’ resource limit

-a

all current limits are reported

-c

the maximum size of core files created

-d

the maximum size of a process’s data segment

-f

the maximum size of files created by the shell

-l

the maximum size a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-p

the pipe buffer size

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number of user processes

-v

the size of virtual memory

If LIMIT is given, it is the new value of the specified resource; the special LIMIT values `soft’, `hard’, and `unlimited’ stand for the current soft limit, the current hard limit, and no limit, respectively. Otherwise, the current value of the specified resource is printed. If no option is given, then -f is assumed. Values are in 1024-byte increments, except for -t, which is in seconds, -p, which is in increments of 512 bytes, and -u, which is an unscaled number of processes.

RLIMIT_AS

The maximum size of the process’s virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2) and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

RLIMIT_CORE

Maximum size of core file. When 0 no core dump files are created. When non-zero, larger dumps are truncated to this size.

RLIMIT_CPU CPU

time limit in seconds. When the process reaches the soft limit, it is sent a SIGXCPU signal. The default action for this signal is to terminate the process. However, the signal can be caught, and the handler can return control to the main program. If the process continues to consume CPU time, it will be sent SIGXCPU once per second until the hard limit is reached, at which time it is sent SIGKILL. (This latter point describes Linux 2.2 through 2.6 behavior. Implementations vary in how they treat processes which continue to consume CPU time after reaching the soft limit. Portable applications that need to catch this signal should perform an orderly termination upon first receipt of SIGXCPU.)

RLIMIT_DATA

The maximum size of the process’s data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

RLIMIT_FSIZE

The maximum size of files that the process may create. Attempts to extend a file beyond this limit result in delivery of a SIGXFSZ signal. By default, this signal terminates a process, but a process can catch this signal instead, in which case the relevant system call (e.g., write(2), truncate(2)) fails with the error EFBIG.

RLIMIT_LOCKS

(Early Linux 2.4 only) A limit on the combined number of flock(2) locks and fcntl(2) leases that this process may establish.

RLIMIT_MEMLOCK

The maximum number of bytes of memory that may be locked into RAM. In effect this limit is rounded down to the nearest multiple of the system page size. This limit affects mlock(2) and mlockall(2) and the mmap(2) MAP_LOCKED operation. Since Linux 2.6.9 it also affects the shmctl(2) SHM_LOCK operation, where it sets a maximum on the total bytes in shared memory segments (see shmget(2)) that may be locked by the real user ID of the calling process. The shmctl(2) SHM_LOCK locks are accounted for separately from the per-process memory locks established by mlock(2), mlockall(2), and mmap(2) MAP_LOCKED; a process can lock bytes up to this limit in each of these two categories. In Linux kernels before 2.6.9, this limit controlled the amount of memory that could be locked by a privileged process. Since Linux 2.6.9, no limits are placed on the amount of memory that a privileged process may lock, and this limit instead governs the amount of memory that an unprivileged process may lock.

RLIMIT_MSGQUEUE

(Since Linux 2.6.8) Specifies the limit on the number of bytes that can be allocated for POSIX message queues for the real user ID of the calling process. This limit is enforced for mq_open(3). Each message queue that the user creates counts (until it is removed) against this limit according to the formula: bytes = attr.mq_maxmsg * sizeof(struct msg_msg *) + attr.mq_maxmsg * attr.mq_msgsize where attr is the mq_attr structure specified as the fourth argument to mq_open(3). The first addend in the formula, which includes sizeof(struct msg_msg *) (4 bytes on Linux/i386), ensures that the user cannot create an unlimited number of zero-length messages (such messages nevertheless each consume some system memory for bookkeeping overhead).

RLIMIT_NICE

(since Linux 2.6.12, but see BUGS below) Specifies a ceiling to which the process’s nice value can be raised using setpriority(2) or nice(2). The actual ceiling for the nice value is calculated as 20 – rlim_cur. (This strangeness occurs because negative numbers cannot be specified as resource limit values, since they typically have special meanings. For example, RLIM_INFINITY typically is the same as -1.)

RLIMIT_NOFILE

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

RLIMIT_NPROC

The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

RLIMIT_RSS

Specifies the limit (in pages) of the process’s resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED.

RLIMIT_RTPRIO

(Since Linux 2.6.12, but see BUGS) Specifies a ceiling on the real-time priority that may be set for this process using sched_setscheduler(2) and sched_setparam(2).

RLIMIT_RTTIME

(Since Linux 2.6.25) Specifies a limit on the amount of CPU time that a process scheduled under a real-time scheduling policy may consume without making a blocking system call. For the purpose of this limit, each time a process makes a blocking system call, the count of its consumed CPU time is reset to zero. The CPU time count is not reset if the process continues trying to use the CPU but is preempted, its time slice expires, or it calls sched_yield(2). Upon reaching the soft limit, the process is sent a SIGXCPU signal. If the process catches or ignores this signal and continues consuming CPU time, then SIGXCPU will be generated once each second until the hard limit is reached, at which point the process is sent a SIGKILL signal. The intended use of this limit is to stop a runaway real-time process from locking up the system.

RLIMIT_SIGPENDING

(Since Linux 2.6.8) Specifies the limit on the number of signals that may be queued for the real user ID of the calling process. Both standard and real-time signals are counted for the purpose of checking this limit. However, the limit is only enforced for sigqueue(2); it is always possible to use kill(2) to queue one instance of any of the signals that are not already queued to the process.

RLIMIT_STACK

The maximum size of the process stack, in bytes. Upon reaching this limit, a SIGSEGV signal is generated. To handle this signal, a process must employ an alternate signal stack (sigaltstack(2)).

ulimit Examples

Turn off core dumps

ulimit -S -c 0

More Reading

• Please see the SYSSTAT Utilities Home for more performance monitoring tools like sar, sadf, mpstat, iostat, pidstat and sa tools.
• Multitasking from the Linux Command Line + Process Prioritization

Man Pages

1. sched_setscheduler
2. cpuset
3. signal
4. getrlimit
5. ulimit
6. ioprio_get
7. ioprio_set

Kernel Documentation

• information on schedstats (Linux Scheduler Statistics)
• real-time group scheduling
• How and why the scheduler’s nice levels are implemented
• information on scheduling domains
• goals, design and implementation of the Complete Fair Scheduler

Future Discussions:

IO Benchmarking: How, Why and With What

Optimizing Servers and Processes for Speed with ionice, nice, ulimit originally appeared on AskApache.com

The first article was meant as a detailed and thorough introduction to speeding up Windows-based PC’s in a way that makes it easy to follow, without getting too specific. So make sure you read that first, and pay the most attention to freeing up RAM, CPU, and Disk IO speed by reducing the number of services and processes that are running, we will deal with defragmentation, hard-drive speed, Disk IO, Prefetching, and Pagefile/Registry Defragmentation now.

This article has some really really really great stuff for ya. It shows which tools (all but one completely free) are the best and you will use them for a long time, they are all very good. That is just a side benefit, as this article is really more of a step-by-step guide to optimizing your system that won’t have to be repeated for at least a year. The result of course is a much more responsive PC.

Whats New

After writing that article I continued my research and testing into the subject on my personal computers. I wanted to test out several additional programs and methods before I wrote about them for you guys, and I found a few really sweet additions that had a very big performance gain for all my computers, from my oldest and slowest PC’s to my new 4K power laptop. This article is primarily focused on optimizing your hard drive data and improving your Disk IO speed, and you will definately see an improvement in speed. It doesn’t get REALLY good until the defragmenting section..

1. Clean Up Hard Drive – Removing unneccessary files
2. Clean Registry – Fixing slow registry problems automatically
3. Ultimate Defragmenting – The best defrag method I use
4. Optimize Physical Hard Disk – Final step that cleans and heals your physical disk

Clean Up Hard Drive

The first step is to clean up all the extra, temporary, and unneccessary files cluttering your hard-drive. The reason is because we will be defragmenting your hard-drive like its never been defragged before, then we are going to go over every single bit and byte of your hard-drive to optimize the physical sectors and storage of your data.. Also we will be running a check of your registry and cleaning out bad links and other slow errors, so get it as clutter-free as possible. DON’T use Windows built-in folder compression, it makes defragmentation worse… DO use 7-zip or winrar to create a solid archive file of any misc directories with a bunch of files… You only need one program to clean your system.

CCleaner

CCleaner is a Small, Fast and Free software that removes unused and temporary files from your system and allows Windows to run faster, more efficiently and gives you more hard disk space. I’ve now been using it for several months and love it. As well as cleaning up old files and settings left by standard Windows components, CCleaner also cleans temporary files and recent file lists for many applications. Including: Firefox, Opera, Safari, Media Player, eMule, Kazaa, Google Toolbar, Netscape, Microsoft Office, Nero, Adobe Acrobat Reader, WinRAR, WinAce, WinZip and more… Google Chrome, Opera, Safari, etc..

• Recycle Bin, Clipboard
• Windows Temporary files, Windows Log files, Chkdsk file fragments
• Recent Documents (on the Start Menu), Run history (on the Start Menu)
• Windows XP Search Assistant history, old Prefetch data, Windows memory dumps after crashes

Download CCleaner

Registry Cleaning

The registry is Windows biggest mistake, (although I’m sure they like it), and basically holds all the information for your programs and Windows. Things like the size of your windows, recent file lists, icon files for different icons, etc.. I’ve never seen a computer that didn’t have some registry issues, so this needs to be cleaned and may have a huge impact on your speed. Real quickly, here are some programs to backup and restore your registry, optimize and defrag your registry, and finally search and clean any errors in your registry.

CCleaner

Yup! CCleaner also takes care of most of the performance issues of your registry. It’s very safe and fast. CCleaner uses an advanced Registry Cleaner to check for problems and inconsistencies. It checks the following:

• ClassIDs, ProgIDs, Application Paths, Icons
• Uninstallers, Shared DLLs, Fonts, Help File references
• File Extensions, ActiveX Controls, Invalid Shortcuts and more…

Download CCleaner

ERUNT – The Emergency Recovery Utility NT

ERUNT is a Registry Backup and Restore utility for Windows NT/2000/2003/XP. I use this to backup my registry automatically or on command.. Ive used it for years and it’s always good to backup before you do anything.

# Here's the command I use (only for advanced users familiar with autoback)
"%ProgramFiles%\ERUNT\AUTOBACK.EXE" %SystemRoot%\ERDNT\#Date# sysreg curuser otherusers /noconfirmdelete /noprogresswindow /days:45 /alwayscreate

Download ERUNT · (Details)

NTGREGOPT – NT Registry Optimizer

NTGREGOPT is a Registry Optimization tool for Windows NT/2000/2003/XP/Vista that minimizes the size of your registry files by simply compacting the registry hives to the minimum size possible.

Registry files in an NT-based system can become fragmented over time, occupying more space on your hard disk than necessary and decreasing overall performance. You should use the NTREGOPT utility regularly, but especially after installing or uninstalling a program, to minimize the size of the registry files and optimize registry access. The program works by recreating each registry hive “from scratch”, thus removing any slack space that may be left from previously modified or deleted keys. It does NOT change the contents of the registry in any way, nor does it physically defrag the registry files on the drive. I recommend using this once every couple weeks. I scheduled it to run automatically.

Download NTGREGOPT · (Details)

Ultimate Defragmenting

I say “Ultimate Defragmenting” because this is the result of a lot of testing of all the various defragmenting software out there, reading a lot of documentation, and running benchmarking to find the fastest results. This is a mix of several individual defragmenting steps combined for a once-a-year ultimate defragmenting session. This is what I use today, and although it’s altogether a long process, each step you’ll add a new tool or skill that you can use by itself from here on out.

ATTENTION: While running MyDefrag/JkDefrag, SpinRite, and UltraDefrag your computer can get very hot and that is very not cool. I set my laptop on a coke can and pointed a small desk fan at it which kept it very very cool, so do what you can to minimize heat during these programs.

PageDefrag

PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your paging files and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files (where system memory is saved when you hibernate a laptop). One of the limitations of the Windows NT/2000 defragmentation interface is that it is not possible to defragment files that are open for exclusive access. Thus, standard defragmentation programs can neither show you how fragmented your paging files or Registry hives are, nor defragment them. Paging and Registry file fragmentation can be one of the leading causes of performance degradation related to file fragmentation in a system.

I personally keep this enabled for every boot, as it only takes a few seconds after the first time it’s run.

Download PageDefrag

MyDefrag

JkDefrag is a disk defragmenter and optimizer for Windows 2000/2003/XP/Vista/2008/X64. Completely automatic and very easy to use, fast, low overhead, with several optimization strategies, and can handle floppies, USB disks, memory sticks, and anything else that looks like a disk to Windows. Included are a Windows version, a commandline version (for scheduling by the task scheduler or for use from administrator scripts), a screensaver version, a DLL library (for use from programming languages), versions for Windows X64, and the complete sources. (Frequently Asked Questions)

After trying out dozens of degragmenting programs, this is my favorite. I utilize the cool screensaver function and just run sometimes when I’m calling it a day.

Tips and tricks

• Many users start looking for defragmentation/optimization programs when their computer becomes slow. The main reason for a slow computer is a full harddisk. A full harddisk is slow because the distance between files is greater than on a fresh practically empty harddisk. Deleting half the data on a full disk will just about double the speed. The more free diskspace, the faster your computer will be.
• Buy a second harddisk (for example an USB harddisk) and move little used stuff from your primary harddisk to that secondary harddisk. The second disk can also be used for backing up the primary disk.
• When buying a new computer, buy the biggest harddisk you can afford. Investing in a bigger harddisk gives more speed-per-dollar than investing in a faster CPU or investing in more memory.
• Cleanup old junk from your harddisk before running MyDefrag. You can clean Windows files with for example “Start -> Programs -> Accessories -> System Tools -> Disk Cleanup”, or with something like the freeware CCleaner program.
• Reboot before running MyDefrag. This will release files that are in use, so they can be defragmented and optimized.
• Boot into Windows safe mode by pressing F8 when booting, and then run MyDefrag. It will be slower because the Windows disk cache is off in safe mode, but MyDefrag will be able to process (a few) more files.
• Stop your real time virus scanner before running MyDefrag. Virus scanners check all disk activity, making defragmentation and optimization very slow.
• Move the swap file to another volume, reboot, defragment, and move the swap file back. If you don’t have a second volume then temporarily make the swap file small, for example 100Mb.
• Package unused files with a packager such as 7-zip. The packagefile not only takes less harddisk space, but will also defragment and optimize much faster than the individual files. Note: This does not apply to Windows NTFS compression, which will actually make defragmentation and optimization slower.
• The first partition on a harddisk is significantly faster than other partitions. Try to use other partitions only for data that is used less often, such as music, movies, archives, backups, logfiles.
• If you have 2 physical harddisks of the same speed, then place the pagefile on the first partition of the second harddisk.

The way I recommend is to run MyDefrag at the highest level of defragmentation once, which took my fastest PC almost 30 hours. Once that is done you can just run it normally in 20 minutes or so.. This software also has the best defrag information I’ve found to date, so check out the documentation on the site.

Download MyDefrag

UltraDefrag

UltraDefrag is a powerful Open Source Defragmentation tool for Windows Platform. It is very fast, because the defragmenting is done via the kernel-mode driver. There are three interface available : Graphical, Console and Native. I personally like the MyDefrag more because I think it does a better job, but I also use the UltraDefrag tool because it has one very important feature like the PageDefrag tool. It has a native version. That means it can run before Windows loads up by utilizing the bootexecute, the same place that windows chkdsk runs at boot. It also can takeover for Windows builtin prefetcher, to speed up the loading of frequently used programs, which I’ll explain a bit later.

I set ultradefrag up after the MyDefrag 30hour defrag completes, to run at boot and control the prefetching, then I erase any prefetch files currently saved and reboot which lets it defrag the system.

erase /Q "%SYSTEMROOT%\Prefetch\*.*"

Once both MyDefrag and UltraDefrag have run THEN I finally login to windows and don’t open any programs to let the windows OS files get optimized by not doing anything at all for 5 minutes.. Then I reboot and log back in and this time I don’t do anything for 30 minutes and reboot. Finally I log back in and this time I instantly load up 10 of my most frequently used programs, (DreamWeaver, Photoshop, Firefox, Chrome, Notepad2, Thunderbird, Internet Explorer, and a few others) and once they are all loaded I don’t do anything for an hour. Then I reboot and repeat that same process.

This may seem odd or made up but I do my research and this allows your prefetched files to be optimized, including your boot prefetch files. Once that is done I reboot and run all the defrags again. Then I reboot and am ready for the last step.

Download UltraDefrag

Optimize Physical Hard Disk

Now at this point the system is defragged and optimized as much as I can get it, but the last step is to run a program to go over every single bit on our hard-drive-disk to keep the drive clean and healthy and its too technical for me to understand, I just know its amazingly cool and I noticed a big change right away.

HD Tune

HD Tune is a fantastic little utility that you can use to benchmark the DISK IO speed of your various drives, internal and external, fixed and USB, firewire, etc.. Other than using it to determine your fastest drives for moving your program files and temps to, I am just including it in this article because it is an awesome program that you will love.

Download HD Tune

SpinRite

This is the last step in this guide, and was the one thing that surprised me the most in terms of how much of a speed improvement I noticed after using it. SpinRite is the most capable, thorough, and reliable utility that has ever been created for the long term maintenance, recovery, and repair of mass storage systems. SpinRite is not a drive defragmenter. SpinRite operates with the drive’s built-in intelligence to reassign and relocate defective sectors without creating file system fragments. Thus, running SpinRite does not create fragments, but neither does it eliminate any that may exist before it was run. Unlike any other disk utility, SpinRite interfaces directly to the hard disk system’s hardware, rather than working through the system’s operating system or BIOS. FAQ.

The way that we use SpinRite in this article is method #4, Drive Maintenance mode, which reads and writes and verifies every single sector and area of your hard drives, and improves the health of your hard drive a lot. Even on my new 4K dell power laptop, this had a noticeable improvement on speed. After running this 20+ hours for my fastest PC, I rebooted, defragged with jkdefrag, and that is the end of this article.

In case you don’t already know . . . What is SpinRite?

SpinRite is a stand-alone DOS application that specializes in the recovery of marginally or completely unreadable hard and floppy disk data, and in the lifetime maintenance of PC mass storage devices. It earned its stripes many years ago by introducing the concept of non-destructive low-level reformatting and sector interleave optimization. Since then its capabilities have continued to broaden until it has become the premiere tool for disk data recovery and magnetic mass storage drive maintenance. Written in assembly language, SpinRite still performs as well on a clunky old 4.77 megahertz PC/XT as on a screaming 333 megahertz Pentium II.

While SpinRite 6.0 is running, you can toggle through seven displays:

Purchase and Download SpinRite

More Reading

• PCNet File Catch – SpinRite 6.0
• SpinRite 6.0 for Linux Users
• Anticipate Drive Problems Early with SpinRite v6.0
• Black Viper – Windows XP Super Tweaks
• Black Viper – Optimize your Services
• Advanced Windows Shell Tutorial
• Batch Files (Scripts) in Windows
• File System Heirarchy
• Steve Gibson discussing Defragmenting

Windows Optimization – Intense Part II originally appeared on AskApache.com

Raw Speed Benefit

This eliminates millions of bandwidth and resource robbing 304 If-Modified-Since requests.

Renaming links vs. Renaming files

On the server my files are named apache.css and apache.js, but in the xhtml I point to them using the names apache-113.css and apache-113.js, after I change the file I just add 1 to the number, and the new file is cached. They are internally redirected to apache.css and apache.js (invisible to the user) The concept is similar to a “shortcut” in windows or a symlink in BSD. The trick is that I never actually rename the files on the server. I just rename them in the html. That means apache-135.css is served from the file apache.css but the browser/cache only see and know about apache-135.css.

XHTML

NOTE: You can do your own investigating of this sites source code and HTTP headers to see this whole system in action

<link href="http://static.askapache.com/z/c/apache-0031.css" rel="stylesheet" type="text/css" />
<script src="http://static.askapache.com/z/j/apache-0031.js" type="text/javascript"></script>

mod_rewrite code for htaccess or httpd.conf

Updated: 10/20/2008

RewriteEngine On
RewriteBase /
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ /z/([a-z]+)/(.+)-([0-9]+)\.(js|css).*\ HTTP/ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .+ /z/%2/%3.%5 [NC,L]

Alternate mod_rewrite code

RewriteEngine On
RewriteBase /
RewriteRule ^([cij]+)(/?[a-z]*)/([a-z]+)-([0-9]+)\.([a-z]+)$ /$1$2/$3.$5 [L]

Ideal Caching Scheme

Ok so you want the xhtml to be the only file that isn’t cached without being validated, its simple to setup your own caching scheme.

Bad Cache information for a file with the ?v=foo hack

This object will be fresh for 1 week. It can be validated with Last-Modified. This object requests that a Cookie be set; this makes it and other pages affected automatically stale; clients must check them upon every request. Because it contains a query (‘?’) in the URL, many caches will not keep this object.

mod_rewrite Fix for Caching Updated Files originally appeared on AskApache.com

I’ve used just about every CMS/Blog/Forum/Trac/Gallery/etc) and really didn’t like a lot of the way they coded… I could use php but I didn’t KNOW php.. so I’ve had to learn php also, and it was tough to learn the advanced class usage and all the other language specific (but similar) constructs for php. It was especially difficult (but fun and challenging) to program so as to be compatible with php4 and php5 (Such is WordPress). But I kept at it, and soon you can decide for yourself what to make of it.

I can code in plenty of languages (bash, lua, windows .bat and vbs, ocaml, big fan of all things shell) and can work my way through C and even sorta somewhat with assembly. Assembly is the hardest, by far, I’m into easy and powerful languages like Python, Javascript, perl, php, ruby, and CGI. I’ve used PHP for a long time to do various things, but never to build software projects like this. Once I noticed WordPress’s core .php files and the excellent programming I wanted to try and learn hot to do it. The WordPress code is some of the best I’ve seen. It appears the way they built it was planned, and not just dreamt up while typing that I can’t help but do. Every time I read through the core code I learn a new trick or very nice way to do something. Those guys are really good, and I think WordPress is going to dominate for a long long time.

The Strategy

The Password Protection (passpro) plugin has a lot of complex stuff going on, especially for a newbie to PHP and WordPress like me, so after refactoring the whole thing at least 5 times I decided to modify my approach, and wrote the AskApache Google 404 Plugin as a way to practice on a simpler piece of code, while at the same time providing a plugin of value. Eventually I stopped thinking I could just code the whole thing in one sit-down with a stream-of-consciousness, and had to instead modularize the code and focus in on each part before moving to the next (I go without a plan because its fun, just not the most productive, but again, I’m not a programmer in the scientific sense.).

So I decided I had to really learn how WordPress Plugins work, filters, hooks, actions, and basically comfortability at reverse-engineering code, (Im a beginner for the last time), and so with the upcoming release of the AskApache Google 404 Plugin I have succeeded in making an incredibly stable plugin. That way I only have to worry about what the aapasspro plugin is doing, instead of trying to fit it into a framework.

AskApache Google 404 Upgrade

I think its rather unusual to develop a nice plugin like this 404 handler merely for the purpose of improving upon another plugin, but hey it worked. As of 08/03/2009 14:06PM EST I have about 1 hour left of finishing touches to release this upgrade. But as you cantell by my badly edited posts, I don’t have a lot of time to myself. An hour here and there is about it. So it could be up to 2 weeks before I actually have the time to commit the release to the repo. On a sidenote, have you checked out Windows 7 News? I’ve been contracted to do some technical work for them and thought they had an excellent site.

But keep in mind, the 404 PLugin is just where I practice for the passpro plugin, which truly does have features that no other software like it has ever had. I understand the technology behind this plugin, and know it would really have a great impact on improving the Web (esp. WordPress) for all of us, I’ve just had to learn how to make it.

AskApache Password Protection

Probably still a couple weeks away, this plugin is the ultimate culmination of apache hackers dreams, at least those on shared servers (who may be interested in learning how to bypass security of said servers).. So this is something I have much too fun with doing what I like to do.. network/protocol-level security. I’ve examined the source code for many software packages that I use or have used to audit a server’s security, and this simple php plugin in most instances can enumerate with accuraccy most of the server’s setup in about a minute. The catch (and the file permission problems I had to find a workaround too) is that this software is launched on the server, not remotely against the server.

Some of the software I examined was whiskers, nessus, nmap, hping, mozilla source, wireshark, ncftp, netcat, etc.. The closest comparison to the socket-level class I’ve hacked together to those is wireshark. Except that wireshark only interprets (captures) the data passing over the wire, while this class does that and in fact sends and receives the data like netcat or nmap. Its really more similar to metasploit, and can easily be used to send hex, binary, ascii, or any type of payload to the remote or local host.

The Upgrades Begin

Well I started working on them a long time ago. Both the Password Protection plugin and the Google 404 plugin needed serious work. And I finally have it all figured out. Essentially I would work on one and finish an upgrade, but I just wasn’t happy with it and I wold start all over again, refactoring the code. So as I put the finishing touches on those 2 plugins keep an eye out. They are major upgrades. I was able to meet all the goals I had for them, and came up with a lot of more improvements during the process.One of the main things I needed was a socket-level class to perform all kinds of checks and tests on. I need this also for my crazy cache plugin, which my blog is currently using , and I have a 2 more really nice pplugins I use that also needed access to a network class. I wrote about what I was doing with fsockopen, and I’ve been improving on that example ever since. I use this class to do some really powerful and exciting stuff, but you’ll see it soon enough. As an indication of ‘getting it right’ for the Password Protection plugin, the plugin will now work on Windows, Apache, IIS, Lighthttpd, and will even work running on a blackberry web server. So now everyone using wordpress can at least get some security()

Many of the the other improvements focus on using the fsockopen class and .htaccess tricks to basically enumerate and discover all the different capabilities of your particular server; That way you can learn about all the features and security that are possible for your specific server, and the securty modules wi8ll be geared for that as well. FINALLY this plugin is going to be stable, and I just cant wait to see how people react when they learn all great capability their Apache-based Server has that they didn’t have a clue about. Its amazing in that sense, and hackers will love theh way it works.. but your server admins will love it even more because its entirely 100% focused on helping you to set your site up (if you have Apache) to keep spammers out, to keep virii-serving robots and their log-hogging exploit requests and CPU/Mem robiing 404 errors off of your servers for real. This will have a noticeable affect to whoever is running the server. As you can tell.. I am pumped!

Apache can only be influenced by the main server configs and by .htaccess files. Not by php, not by perl, and the main configs are almost never accessible to the masses. But .htaccess files are. And many hosting providers allow and enable .htaccess files, a configuration file for your web server. The advanced features and capabilities of Apache were out of reach for most of us, it just wasn’t possible to enumerate or access, and most hosting providers are infamous for their lack of .htaccess (customer) support. This plugin goes around those problems to give the power back to the people.

y creating custom .htaccess files containing unpublished .htaccess tricks and techniques and combining that with the use of socket-level networking from WordPress (PHP) using fsockopen, we can effectively enumerate and discover an incredible amount of features and settings you will be able to control and use with this plugin.

Here are a few examples of the capabilities of this plugin, some of which I believe no other software can do.. (Open source free to copy!).

1. Current Version of Apache (Down to the API Version)
2. List of ALL Modules currently enabled by Apache (Such as Mod_Rewrite)
3. List of ALL Directives enabled by EACH enabled Module.
4. Enumerate .htaccess Overrides, Context Permissions
5. Test for any builtin Handlers (like the status handler screenshot)
6. Configure SSI (http://www.askapache.com/htaccess/advanced-htaccess-ssi.html#htaccess-ssi-security)

March 1, 2009
I would focus on the method that WordPress uses. The code they have now (2.8 bleeding-edge) still isn’t where it needs to be, but this is some difficult stuff and they have a brilliant start, it’ll work.. just a question of when.

The main issue with the password protection plugin working for some people and not others is due to file permission configurations. The plugin attempts to write/modify files in your blog’s root directory.

November 05, 2008
To make a long story short, I downloaded each major release of the apache httpd source code starting at version 1.3.0 and finishing with version 2.2.11, I then compiled each version and built a HTTPD from source for all these apache versions.

• 1.3.0
• 1.3.1
• 1.3.11
• 1.3.12
• 1.3.14
• 1.3.17
• 1.3.19
• 1.3.2
• 1.3.20
• 1.3.22
• 1.3.23
• 1.3.24
• 1.3.27
• 1.3.28

• 1.3.29
• 1.3.3
• 1.3.31
• 1.3.32
• 1.3.33
• 1.3.34
• 1.3.35
• 1.3.36
• 1.3.37
• 1.3.39
• 1.3.4
• 1.3.41
• 1.3.6
• 1.3.9

• 2.0.35
• 2.0.36
• 2.0.39
• 2.0.40
• 2.0.42
• 2.0.43
• 2.0.44
• 2.0.45
• 2.0.46
• 2.0.47
• 2.0.48
• 2.0.49
• 2.0.50
• 2.0.51

• 2.0.52
• 2.0.53
• 2.0.54
• 2.0.55
• 2.0.58
• 2.0.59
• 2.0.61
• 2.0.63
• 2.1.3-beta
• 2.1.6-alpha
• 2.1.7-beta
• 2.1.8-beta
• 2.1.9-beta

• 2.2.0
• 2.2.10
• 2.2.2
• 2.2.3
• 2.2.4
• 2.2.6
• 2.2.8
• 2.2.9
• 2.2.10
• 2.2.11

Then I went through each version and determined the compatible modules for that version, and I’m pretty confident that I was also able to find each and every directive allowed by the compatible modules for that version (including core directives). See .htaccess directive list. Basically I can now test a server using a variety of methods and determine almost 100% accurately what version of Apache (down to the API) is running, what modules (and versions) are enabled, and each and every directive that is allowed or disallowed for that version. So this is so awesome because now we can enable all sorts of additional security features.

Htaccess enabled Modules

Here are most of the modules that come with Apache. Each one can have new commands that can be used in .htaccess file scopes.

mod_actions, mod_alias, mod_asis, mod_auth_basic, mod_auth_digest, mod_authn_anon, mod_authn_dbd, mod_authn_dbm, mod_authn_default, mod_authn_file, mod_authz_dbm, mod_authz_default, mod_authz_groupfile, mod_authz_host, mod_authz_owner, mod_authz_user, mod_autoindex, mod_cache, mod_cern_meta, mod_cgi, mod_dav, mod_dav_fs, mod_dbd, mod_deflate, mod_dir, mod_disk_cache, mod_dumpio, mod_env, mod_expires, mod_ext_filter, mod_file_cache, mod_filter, mod_headers, mod_ident, mod_imagemap, mod_include, mod_info, mod_log_config, mod_log_forensic, mod_logio, mod_mem_cache, mod_mime, mod_mime_magic, mod_negotiation, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_ftp, mod_proxy_http, mod_rewrite, mod_setenvif, mod_speling, mod_ssl, mod_status, mod_substitute, mod_unique_id, mod_userdir, mod_usertrack, mod_version, mod_vhost_alias

Debugging HTTP protocol

Check this out! I’m particularly happy about this feature, which outputs an exact trace of any requests made by the plugin (such as during the testing phase) by saving the actual raw data sent out on the wire using fsockopen, RX and TX. This is useful for a number of reasons, viewing your headers, finding Redirect Loops, testing RewriteRules, and following the request hop-by-hop for debugging. The below example shows 2 requests for 2 URIs. The first URI is protected using Digest Authentication, the 2nd shows Basic.

 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Digest realm="do or die", nonce="03328f3ec7c7b", algorithm=MD5, domain="/", qop="auth"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 882
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/index.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Digest username="test",realm="do or die",nonce="03328f3ec7c7b",uri="/htaccess/index.txt?testing=query",
cnonce="82d057852a9dc497",nc=00000001,algorithm=MD5,response="9d476e9ea3",qop="auth"
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Authentication-Info: rspauth="9051b01ee26dd62b3e2b40dada694f45", cnonce="82d057852a9dc497", nc=00000001, qop=auth
Last-Modified: Tue, 21 Jul 2009 23:56:00 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````
 
 ______________
|  RAW TRACE   |
==================================================================================================================================
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
 
HTTP/1.1 401 Authorization Required
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
WWW-Authenticate: Basic realm="Po Pimping"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 878
Connection: close
Content-Type: text/html; charset=UTF-8
 
GET /htaccess/po.txt?testing=query HTTP/1.1
Host: www.askapache.com
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1) AA_PassPro/1.9 (http://www.askapache.com/)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Connection: close
Referer: http://www.askapache.com/
Authorization: Basic adfAGAltcA==
 
HTTP/1.1 200 OK
Date: Wed, 22 Jul 2009 06:29:58 GMT
Server: Apache
Last-Modified: Wed, 22 Jul 2009 05:54:39 GMT
Accept-Ranges: bytes
Cache-Control: max-age=3600
Expires: Wed, 22 Jul 2009 07:29:58 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 27
Connection: close
Content-Type: text/plain; charset=UTF-8
```````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````````

.htaccess Directives

AcceptFilter, AcceptMutex, AcceptPathInfo, AccessFileName, Action, AddAlt, AddAltByEncoding, AddAltByType, AddCharset, AddDefaultCharset, AddDescription, AddEncoding, AddHandler, AddIcon, AddIconByEncoding, AddIconByType, AddInputFilter, AddLanguage, AddModuleInfo, AddOutputFilter, AddOutputFilterByType, AddType, Alias, AliasMatch, AllowCONNECT, AllowEncodedSlashes, AllowOverride, Anonymous, Anonymous_Authoritative, Anonymous_LogEmail, Anonymous_MustGiveEmail, Anonymous_NoUserID, Anonymous_NoUserId, Anonymous_VerifyEmail, AssignUserId, AuthAuthoritative, AuthBasicAuthoritative, AuthBasicProvider, AuthDBDUserPWQuery, AuthDBDUserRealmQuery, AuthDBM, AuthDBMAuthoritative, AuthDBMGroupFile, AuthDBMType, AuthDBMUserFile, AuthDefaultAuthoritative, AuthDigestAlgorithm, AuthDigestDomain, AuthDigestFile, AuthDigestGroupFile, AuthDigestNcCheck, AuthDigestNonceFormat, AuthDigestNonceLifetime, AuthDigestProvider, AuthDigestQop, AuthDigestShmemSize, AuthGroupFile, AuthLDAPAuthzEnabled, AuthLDAPBindDN, AuthLDAPBindON, AuthLDAPBindPassword, AuthLDAPCharsetConfig, AuthLDAPCompareDNOnServer, AuthLDAPDereferenceAliases, AuthLDAPEnabled, AuthLDAPFrontPageHack, AuthLDAPGroupAttribute, AuthLDAPGroupAttributeIsDN, AuthLDAPRemoteUserAttribute, AuthLDAPRemoteUserIsDN, AuthLDAPStartTLS, AuthLDAPURL, AuthLDAPUrl, AuthName, AuthType, AuthUserFile, AuthzDBMAuthoritative, AuthzDBMType, AuthzDefaultAuthoritative, AuthzGroupFileAuthoritative, AuthzLDAPAuthoritative, AuthzOwnerAuthoritative, AuthzUserAuthoritative, BS2000Account, BalancerMember, BrowserMatch, BrowserMatchNoCase, BufferedLogs, CGIMapExtension, CacheDefaultExpire, CacheDirLength, CacheDirLevels, CacheDisable, CacheEnable, CacheExpiryCheck, CacheFile, CacheForceCompletion, CacheGcClean, CacheGcDaily, CacheGcInterval, CacheGcMemUsage, CacheGcUnused, CacheIgnoreCacheControl, CacheIgnoreHeaders, CacheIgnoreNoLastMod, CacheLastModifiedFactor, CacheMaxExpire, CacheMaxFileSize, CacheMaxStreamingBuffer, CacheMinFileSize, CacheNegotiatedDocs, CacheRoot, CacheSize, CacheStoreNoStore, CacheStorePrivate, CacheTimeMargin, CharsetDefault, CharsetOptions, CharsetSourceEnc, CheckCaseOnly, CheckSpelling, ChildPerUserId, ContentDigest, CookieDomain, CookieExpires, CookieLog, CookieName, CookieStyle, CookieTracking, CoreDumpDirectory, CustomLog, DAV, DAVDepthInfinity, DAVGenericLockDB, DAVMinTimeout, DBDExptime, DBDKeep, DBDMax, DBDMin, DBDParams, DBDPersist, DBDPrepareSQL, DBDriver, Dav, DavDepthInfinity, DavGenericLockDB, DavLockDB, DavMinTimeout, DefaultIcon, DefaultLanguage, DefaultType, DeflateBufferSize, DeflateCompressionLevel, DeflateFilterNote, DeflateMemLevel, DeflateWindowSize, Directory, DirectoryIndex, DirectoryMatch, DirectorySlash, DocumentRoot, DumpIOInput, DumpIOOutput, EnableExceptionHook, EnableMMAP, EnableSendfile, ErrorDocument, ErrorLog, Example, ExpiresActive, ExpiresByType, ExpiresDefault, ExtFilterDefine, ExtFilterOptions, ExtendedStatus, FancyIndexing, FileETag, Files, FilesMatch, FilterChain, FilterDeclare, FilterProtocol, FilterProvider, FilterTrace, ForceLanguagePriority, ForceType, ForensicLog, GprofDir, GracefulShutdownTimeout, Group, Header, HeaderName, HostNameLookups, HostnameLookups, ISAIPFakeAsync, ISAPIAppendLogToErrors, ISAPIAppendLogToQuery, ISAPICacheFile, ISAPIFakeAsync, ISAPILogNotSupported, ISAPIReadAheadBuffer, IdentityCheck, IdentityCheckTimeout, IfDefine, IfModule, IfVersion, ImapBase, ImapDefault, ImapMenu, Include, IndexIgnore, IndexOptions, IndexOrderDefault, IndexStyleSheet, KeepAlive, KeepAliveTimeout, LDAPCacheEntries, LDAPCacheTTL, LDAPCertDBPath, LDAPConnectionTimeout, LDAPOpCacheEntries, LDAPOpCacheTTL, LDAPSharedCacheFile, LDAPSharedCacheSize, LDAPTrustedClientCert, LDAPTrustedGlobalCert, LDAPTrustedMode, LDAPVerifyServerCert, LanguagePriority, Limit, LimitExcept, LimitInternalRecursion, LimitRequestBody, LimitRequestFields, LimitRequestFieldsize, LimitRequestLine, LimitXMLRequestBody, Listen, ListenBacklog, LoadFile, LoadModule, Location, LocationMatch, LockFile, LogFormat, LogLevel, MCacheMaxObjectCount, MCacheMaxObjectSize, MCacheMaxStreamingBuffer, MCacheMinObjectSize, MCacheRemovalAlgorithm, MCacheSize, MMapFile, MaxClients, MaxKeepAliveRequests, MaxMemFree, MaxRequestsPerChild, MaxSpareServers, MaxSpareThreads, MaxSpareThreadsPerChild, MaxThreads, MetaDir, MetaFiles, MetaSuffix, MimeMagicFile, MinSpareServers, MinSpareThreads, ModMimeUsePathInfo, MultiviewsMatch, NWSSLTrustedCerts, NWSSLUpgradeable, NameVirtualHost, NoProxy, NumServers, Options, PassEnv, PerlAccessHandler, PerlAuthenHandler, PerlAuthzHandler, PerlChildExitHandler, PerlChildInitHandler, PerlCleanupHandler, PerlDispatchHandler, PerlFixupHandler, PerlFreshRestart, PerlHandler, PerlHeaderParserHandler, PerlInitHandler, PerlLogHandler, PerlModule, PerlPassEnv, PerlPostReadRequestHandler, PerlRequire, PerlRestartHandler, PerlSendHeader, PerlSetEnv, PerlSetVar, PerlSetupEnv, PerlTaintCheck, PerlTransHandler, PerlTypeHandler, PerlWarn, PidFile, Port, Protocol, ProtocolEcho, Proxy, ProxyBadHeader, ProxyBlock, ProxyDomain, ProxyErrorOverride, ProxyFtpDirCharset, ProxyIOBufferSize, ProxyMatch, ProxyMaxForwards, ProxyPass, ProxyPassInterpolateEnv, ProxyPassMatch, ProxyPassReverse, ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath, ProxyPreserveHost, ProxyReceiveBufferSize, ProxyRemote, ProxyRemoteMatch, ProxyRequests, ProxySet, ProxyStatus, ProxyTimeout, ProxyVia, RLimitCPU, RLimitMEM, RLimitNPROC, ReadmeName, Redirect, RedirectMatch, RedirectPermanent, RedirectTemp, RemoveCharset, RemoveEncoding, RemoveHandler, RemoveInputFilter, RemoveLanguage, RemoveOutputFilter, RemoveType, RequestHeader, Require, RewriteBase, RewriteCond, RewriteEngine, RewriteLock, RewriteLog, RewriteLogLevel, RewriteMap, RewriteOptions, RewriteRule, SSIAccessEnable, SSIEndTag, SSIErrorMsg, SSIStartTag, SSITimeFormat, SSIUndefinedEcho, SSLCACertificateFile, SSLCACertificatePath, SSLCADNRequestFile, SSLCADNRequestPath, SSLCARevocationFile, SSLCARevocationPath, SSLCertificateChainFile, SSLCertificateFile, SSLCertificateKeyFile, SSLCipherSuite, SSLCryptoDevice, SSLEngine, SSLHonorCipherOrder, SSLLog, SSLLogLevel, SSLMutex, SSLOptions, SSLPassPhraseDialog, SSLProtocol, SSLProxyCACertificateFile, SSLProxyCACertificatePath, SSLProxyCARevocationFile, SSLProxyCARevocationPath, SSLProxyCipherSuite, SSLProxyEngine, SSLProxyMachineCertificateFile, SSLProxyMachineCertificatePath, SSLProxyProtocol, SSLProxyVerify, SSLProxyVerifyDepth, SSLRandomSeed, SSLRequire, SSLRequireSSL, SSLSessionCache, SSLSessionCacheTimeout, SSLUserName, SSLVerifyClient, SSLVerifyDepth, Satisfy, ScoreBoardFile, Script, ScriptAlias, ScriptAliasMatch, ScriptInterpreterSource, ScriptLog, ScriptLogBuffer, ScriptLogLength, ScriptStock, SecureListen, SendBufferSize, ServerAdmin, ServerAlias, ServerLimit, ServerName, ServerPath, ServerRoot, ServerSignature, ServerTokens, SetEnv, SetEnvIf, SetEnvIfNoCase, SetHandler, SetInputFilter, SetOutputFilter, StartServers, StartThreads, Substitute, SuexecUserGroup, ThreadLimit, ThreadStackSize, ThreadsPerChild, TimeOut, Timeout, TraceEnable, TransferLog, TypeAuthDBMUserFile, TypesConfig, UnsetEnv, UseCanonicalName, UseCanonicalPhysicalPort, User, UserDir, VirtualDocumentRoot, VirtualDocumentRootIP, VirtualHost, VirtualScriptAlias, VirtualScriptAliasIP, Win32DisableAcceptEx, XBitHack, allow, deny, order, php_admin_flag, php_admin_value, php_flag, php_value

You can view the plugins home page, old, or view it on the wordpress.org site.

An AskApache Plugin Upgrade to Rule them All originally appeared on AskApache.com

Round Robin Concept

A few months ago I was given a free hosting account on HostGator to evaluate and tempt me away from DreamHost to become a Gator. I get a lot of these types of offers from time to time from the smaller Web companies who read AskApache.. but when I noticed the SSH access was jailed and saw how restrictive the shell was I felt like I was on a windows box.. I want a shell, cpanel sucks. I compile and run everything from the shell so thats was a deal-breaker and I sorta forgot all about it.

The goal is to add the HostGator server to be an exact mirror of the static.askapache.com domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.

Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)

DNS – Domain Name System

The Domain Name System is often analogized as a “phone book” for the Internet because it translates human-friendly computer hostnames into IP addresses. For example, www.askapache.com translates to 208.113.134.190. Every request for a human-friendly hostname first needs to be translated to the IP before the server can be queried. Normally each hostname corresponds to exactly 1 IP address. But in a Round Robin setup, the idea is to have the hostname correspond to multiple IP addresses, which are different servers that house the exact same content, resulting in some hosts requesting files from one server, and another host requesting files from the other server, resulting in less CPU resources and bandwidth.

Here is an the zone for static.askapache.com Round Robin records:

Round Robin DNS

Round robin DNS is a technique of load distribution, load balancing, or fault-tolerance provisioning multiple, redundant Internet Protocol service hosts, e.g., Web servers, FTP servers, by managing the Domain Name System’s (DNS) responses to address requests from client computers according to an appropriate statistical model.

In its simplest implementation Round-robin DNS works by responding to DNS requests not only with a single IP address, but a list of IP addresses of several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query so that on different connection attempts clients would receive service from different providers, thus distributing the overall load among servers.

Round robin DNS is often used for balancing the load of geographically-distributed Web servers. For example, a company has one domain name and three identical web sites residing on three servers with three different IP addresses. When one user accesses the home page it will be sent to the first IP address. The second user who accesses the home page will be sent to the next IP address, and the third user will be sent to the third IP address. In each case, once the IP address is given out, it goes to the end of the list. The fourth user, therefore, will be sent to the first IP address, and so forth.

Although easy to implement, round robin DNS has problematic drawbacks, such as those arising from record caching in the DNS hierarchy itself, as well as client-side address caching and reuse, the combination of which can be difficult to manage. Round robin DNS should not solely be relied upon for service availability. If a service at one of the addresses in the list fails, the DNS will continue to hand out that address and clients will still attempt to reach the inoperable service.

Does Round Robin Work

Definately. I can look at the access logs for both servers and see that they are splitting the requests nicely. It is definately not an exact split however, look at these statistics and you’ll see what I mean.

$ dig @ns1.dreamhost.com +authority +all static.askapache.com
 
;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       69.56.174.114
static.askapache.com.        14400   IN      A       64.111.114.111
 
$ dig @ns1.dreamhost.com +authority +all static.askapache.com
 
;; ANSWER SECTION:
static.askapache.com.        14400   IN      A       64.111.114.111
static.askapache.com.        14400   IN      A       69.56.174.114

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used which has the effect of significantly increasing the load on the DNS (and is not always implemented consistently). In this case the cure may be worse than the disease Good news we have good load balancing on our web servers. Bad news we need 17 more DNS servers!. Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static list, loads should be reasonably balanced .

Full root server query

Tracing to static.askapache.com[a] via A.ROOT-SERVERS.NET, maximum of 3 retries
A.ROOT-SERVERS.NET [.] (198.41.0.4)
 |\___ L.GTLD-SERVERS.NET [com] (192.41.162.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ D.GTLD-SERVERS.NET [com] (192.31.80.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ J.GTLD-SERVERS.NET [com] (192.48.79.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ H.GTLD-SERVERS.NET [com] (192.54.112.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ E.GTLD-SERVERS.NET [com] (192.12.94.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ F.GTLD-SERVERS.NET [com] (192.35.51.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ I.GTLD-SERVERS.NET [com] (192.43.172.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ G.GTLD-SERVERS.NET [com] (192.42.93.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ B.GTLD-SERVERS.NET [com] (2001:0503:231d:0000:0000:0000:0002:0030) Not queried
 |\___ B.GTLD-SERVERS.NET [com] (192.33.14.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ A.GTLD-SERVERS.NET [com] (2001:0503:a83e:0000:0000:0000:0002:0030) Not queried
 |\___ A.GTLD-SERVERS.NET [com] (192.5.6.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ C.GTLD-SERVERS.NET [com] (192.26.92.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 |\___ M.GTLD-SERVERS.NET [com] (192.55.83.30)
 |     |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
 |     |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
 |      \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
  \___ K.GTLD-SERVERS.NET [com] (192.52.178.30)
       |\___ ns3.dreamhost.com [askapache.com] (66.33.216.216) Got authoritative answer
       |\___ ns2.dreamhost.com [askapache.com] (208.96.10.221)
        \___ ns1.dreamhost.com [askapache.com] (66.33.206.206) Got authoritative answer
 
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 64.111.114.111
  ns1.dreamhost.com (66.33.206.206)       static.askapache.com -> 69.56.174.114
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 64.111.114.111
  ns3.dreamhost.com (66.33.216.216)       static.askapache.com -> 69.56.174.114

Live Online DNS Tools

• DNSstuff.com – Your Destination for DNS and Networking Tools
• ZoneCheck
• Quick Check – Pingability.com
• Squishywishywoo: complete dns traversal checking
• ZoneCut DNS
• DNS Sleuth
• DNS test – Domain name DNS test – pweb.cz
• OpenDNS – Support – CacheCheck
• DNSReport
• DNSTrace
• intoDNS
• DNSthru.com
• DNS Doctor
• Power Check

More Reading

• HOWTO – Configure Load Balancing
• Load Sharing with DNS

RFC’s related to DNS

• RFC 920: Specified original TLDs: .arpa, .com, .edu, .org, .gov, .mil and two-character country codes
• RFC 1032: Domain administrators guide
• RFC 1033: Domain administrators operations guide
• RFC 1034: Domain Names – Concepts and Facilities.
• RFC 1035: Domain Names – Implementation and Specification
• RFC 1101: DNS Encodings of Network Names and Other Types
• RFC 1123: Requirements for Internet Hosts — Application and Support
• RFC 1912: Common DNS Operational and Configuration Errors
• RFC 1995: Incremental Zone Transfer in DNS
• RFC 1996: A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)
• RFC 2136: Dynamic Updates in the domain name system (DNS UPDATE)
• RFC 2181: Clarifications to the DNS Specification
• RFC 2182: Selection and Operation of Secondary DNS Servers
• RFC 2308: Negative Caching of DNS Queries (DNS NCACHE)
• RFC 2317: Classless IN-ADDR.ARPA delegation
• RFC 2671: Extension Mechanisms for DNS (EDNS0)
• RFC 3597: Handling of Unknown DNS Resource Record (RR) Types
• RFC 3696: Application Techniques for Checking and Transformation of Names
• RFC 4343: Domain Name System (DNS) Case Insensitivity Clarification
• RFC 4592: The Role of Wildcards in the Domain Name System
• RFC 4892: Requirements for a Mechanism Identifying a Name Server Instance
• RFC 5001: DNS Name Server Identifier Option (NSID)
• RFC 5395: Domain Name System (DNS) IANA Considerations

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Table of Contents

• Round Robin Concept
• DNS – Domain Name System
• Round Robin DNS
• Efficacy of DNS Load Balancing
• Live Online DNS Tools
• More Reading
• RFC’s related to DNS

DNS Round Robin Configuration using Rsync over SSH originally appeared on AskApache.com

1. Improved Directory Listings and SEO
2. Enhanced Error Pages
3. Dealing with Spam and Website Attacks

Improved Directory Listings and SEO

One way that I use SSI is for improved Directory Listing. Anyone here like SEO? Check out my WordPress uploads directory to get an idea. Basically I can customize the header and footer of each directory.. Pretty sweet, thanks Apache!

In your /uploads/.htaccess

# turn on auto-indexing and turn off SSI's ability to exec
Options None
Options SymLinksIfOwnerMatch Indexes IncludesNOEXEC
 
# we need to make sure files are displayed when requested, not executed or parsed
AddType text/plain .ini .sh .bsh .bash .csh .var .asc .md5 .sha .sha1 .cgi .pl .php .inc .asp .exe .bin
DefaultType text/plain
 
# turn on auto-indexing, with askapache-optimized options
IndexOptions FancyIndexing SuppressColumnSorting SuppressHTMLPreamble IconHeight=22 IconWidth=20
IndexOptions IgnoreClient NameWidth=40 DescriptionWidth=* XHTML FoldersFirst
 
# don't show these files and folders
IndexIgnore .htaccess .ht* *_notes *.log feed inc HEADER.html FOOTER.html feed*.gif
 
# the SSI files used for the header and footer
HeaderName /ssi/HEADER.html
ReadmeName /ssi/FOOTER.html
 
# used to determine the time and for SSI output
SetEnv TZ America/Indianapolis
SetEnv SERVER_ADMIN webmaster@askapache.com

In your /ssi/.htaccess

# makes files ending in .html be filtered through the INCLUDES filter before being sent to client
AddOutputFilter Includes html

My HEADER.html

<!--#set var="PAGETITLE" value="-- static.askapache.com" -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
  <title><!--#echo encoding="none" var="REQUEST_URI" --> <!--#echo encoding="none" var="PAGETITLE" --></title>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <link rel="stylesheet" href="http://static.askapache.com/c/error.css" />
  <link rev="made" href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->" />
</head>
<body>
<h1><a href="http://www.askapache.com/" title="AskApache Home" accesskey="1">
<img src="http://www.askapache.com/nlogo.jpg" height="75" alt="AskApache" />
</a> <!--#echo encoding="url" var="REQUEST_URI" --> <!--#echo encoding="none" var="PAGETITLE" --></h1>
<hr />

My FOOTER.html

<p>Find the information you are looking for on the <a href="http://www.askapache.com/">AskApache Home page</a>
or the <a href="http://www.askapache.com/search<!--#echo encoding="url" var="REQUEST_URI" -->">
AskApache search page</a>.</p>
<hr />
<address>
  <small>$Id:<!--#echo encoding="none" var="UNIQUE_ID" --> E:<!--#echo encoding="none" var="REDIRECT_STATUS" -->,v 1.30
<!--#config timefmt="%c" --><!--#echo var="DATE_LOCAL" --></small><br />
  <small><!--#echo var="SERVER_SOFTWARE" --></small><br /><br />
  static.askapache.com -- AskApache | <a href="http://www.askapache.com/about/">Webmaster</a>
| Copyright &copy; 2009 AskApache<br />
</address>
<!--#if expr="$REMOTE_ADDR = 10.10.10.10" -->
<pre ><!--#printenv --></ pre>
<!--#endif -->
<script src="http://static.askapache.com/j/apache-0780.js" type="text/javascript"></script>
<script type="text/javascript">var pageTracker = _gat._getTracker("UA-7"+"321"+"53-38");
pageTracker._initData();pageTracker._trackPageview();</script>
</body>
</html>

Enhanced Error Pages

If you are using WordPress, I’m sure you are using my AskApache Google 404 Plugin, but whatever your error pages may be, using SSI you can make them better. My 404 Error Page is handled by WordPress and PHP, but all of my other error pages (codes 4xx to 5xx) are handled using SSI. You can check out any of them: /show-error-400, /show-error-403, /show-error-503, etc..

Notice the email note which has the subject prefilled? Thats one of the main uses for SSI, you can add forms to your errordocuments and get notified of problems which mean you can fix them.

Add this to your /.htaccess for each ErrorDocument you make.

ErrorDocument 503 /errordocs/503.html

My /errordocs/.htaccess

# turn on symlinks for rewrites and turn off SSI's ability to exec
Options None
Options SymLinksIfOwnerMatch IncludesNOEXEC
 
# makes files ending in .html be filtered through the INCLUDES filter before being sent to client
AddOutputFilter Includes html
 
# this internal apache variable prevents your errordocs from allowing keep-alive connections
SetEnv nokeepalive
 
# used to determine the time and for SSI output
SetEnv TZ America/Indianapolis
SetEnv SERVER_ADMIN webmaster@askapache.com

My /errordocs/503.html

<!--#set var="TITLE" value="Service Temporarily Unavailable" -->
<!--#include virtual="/errordocs/TOP.html" -->
<p>The server is temporarily unable to service your
request due to <strong>maintenance downtime</strong>
or super-crazy-extreme capacity problems. Please try
again later... Or <a href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->">send me an email</a> and let me know about it..</p>
<!--#include virtual="/errordocs/BOTTOM.html" -->

My /errordocs/TOP.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
  <title><!--#echo encoding="none" var="REQUEST_URI" --> <!--#echo encoding="none" var="TITLE" --></title>
  <meta http-equiv="content-type" content="text/html; charset=UTF-8" />
  <link rel="stylesheet" href="http://static.askapache.com/c/error.css" />
  <link rev="made" href="mailto:<!--#echo encoding="url" var="SERVER_ADMIN" -->" />
</head>
<body>
<h1><a href="http://www.askapache.com/" title="AskApache Home" accesskey="1"><img src="http://www.askapache.com/nlogo.jpg" height="75" alt="AskApache" />
</a> <!--#echo encoding="url" var="REQUEST_URI" --> <!--#echo encoding="none" var="TITLE" --></h1>
<hr />

My /errordocs/BOTTOM.html

<p>If this should not be an error please <a href="mailto:webmaster@askapache.com?subject=ID#<!--#echo encoding="none" var="UNIQUE_ID" -->">email</a> me right away.</p>
<!--#if expr="$HTTP_REFERER" -->
<p>You came from <!--#echo var="HTTP_REFERER"--></p>
<!--#endif -->
<p>If you still have a question, please try to find the information you are looking
for on the <a href="http://www.askapache.com/">AskApache Home page</a> or
the <a href="http://www.askapache.com/search/">AskApache search page</a>.</p>
<hr />
<address>
  <a href="http://www.askapache.com/about/">Webmaster</a>
  $Id:<!--#echo encoding="none" var="UNIQUE_ID" --> Error-<!--#echo encoding="none" var="REDIRECT_STATUS" -->,v 1.30 <!--#config timefmt="%c" --><!--#echo var="DATE_LOCAL" --><br />
  <!--#echo var="SERVER_SOFTWARE" -->
</address>
<!--#if expr="$REMOTE_ADDR = 10.10.10.10" -->
<pre ><!--#printenv --></ pre>
<!--#endif -->
</body>
</html>

Dealing with Spam and Website Attacks

Say you are using some nice .htaccess rewrite rules to block offending bots, web scrapers, and other nefarious net characters. Instead of just sending a 403 Denied, you could send them to be handled by an SSI document that could do any number of things.. From adding the offending bot’s IP address to the .htaccess Deny List (blacklisting), emailing you with an alert, emailing the IP Block Owner, executing a denial-of-service response ala guardian (script will return an artificially high Content-Length, and will then spoon-feed content bytes back to the client at a rate of one byte per second, for single-threaded or fixed-threadpool clients, this will hang all of their requests and render the attack inoperable), or just output a frightening looking message which usually does the trick if the bot is humanoid.

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\=?(http|ftp|ssl|https):/.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\?.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\.(asp|ini|dll).*\ HTTP/ [NC]
RewriteRule .* - [F]
 
ErrorDocument 403 /errordocs/f-off.html

Example /errordocs/f-off.html View it Live (Not Responsible for mental or physical harm caused by fright)

<html><head><title>WARNING <!--#echo encoding="none" var="REMOTE_ADDR" -->
- SECURITY TEAM ALERTED</title></head><body>
<h2>ATTENTION <!--#echo encoding="none" var="REMOTE_ADDR" --></h2>
<p>You have been flagged by our system as a potential threat.</p>
<p>This request has been logged and the Security Team has been notified.<br>
Repeated attempts from IP address <strong><!--#echo encoding="none" var="REMOTE_ADDR" --></strong> will result <br>
in the IP automatically being <strong>blacklisted</strong> on this server<br>
and sent to blacklists around the world.  Additionally, the Security Team<br>
WILL alert the IP Address Block Owner and/or contact the authorities.<br><br></p>
 
<h3>LOGGED INFORMATION FROM THIS REQUEST</h3>
< pre>
REMOTE_ADDR   :  <!--#echo encoding="none" var="REMOTE_ADDR" -->
REMOTE_PORT   :  <!--#echo encoding="none" var="REMOTE_PORT" -->
REQUEST_METHOD:  <!--#echo encoding="none" var="REQUEST_METHOD" --><!--#if expr="$HTTP_REFERER"
-->
REFERER       :  <!--#echo encoding="url" var="HTTP_REFERER" --><!--#endif
-->
REQUEST_URI   :  <!--#echo encoding="url" var="REQUEST_URI" --><!--#if expr="$HTTP_USER_AGENT"
-->
USER_AGENT    :</ pre>
< pre style="white-space:normal;margin-left:150px;max-width:80%;">  <!--#echo encoding="none" var="HTTP_USER_AGENT" -->
</ pre>
< pre><!--#endif
--><!--#if expr="$HTTP_COOKIE"
-->
COOKIES       :</ pre>
< pre style="white-space:normal;margin-left:150px;max-width:80%;">  <!--#echo encoding="none" var="HTTP_COOKIE" -->
</ pre>
< pre><!--#endif
--><!--#if expr="$QUERY_STRING"
-->
QUERY_STRING  :  <!--#echo encoding="url" var="QUERY_STRING" --><!--#endif
-->
</ pre></body></html>

Server Side Includes Detailed Info

Ok now that we have the real-world usage out of the way, lets dig in a bit to the actual module mod_include, which if you want you can view the source code here.

Enabling Server-Side Includes

Server Side Includes are implemented by the INCLUDES filter. For backwards compatibility, the server-parsed handler also activates the INCLUDES filter. As well, Apache will activate the INCLUDES filter for any document with mime type text/x-server-parsed-html or text/x-server-parsed-html3 (and the resulting output will have the mime type text/html). If documents containing server-side include directives are given the extension .shtml, the following directives will make Apache parse them and assign the resulting document the mime type of text/html:

AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

The following directive must be given for the directories containing the shtml files (typically in a section, but this directive is also valid in .htaccess files if AllowOverride Options is set):

Options +Includes

Server-Side Include Directives

These are the Directives allowed in .htaccess files that are handled by mod_include. Note that other modules may add additional directives, for instance the exec SSI Directive is supplied by mod_cgi. This is how mod_cgi registers with mod_include to provide processing of the exec directive. This is the code required to handle the “exec” SSI directive.

cgi_pfn_reg_with_ssi = APR_RETRIEVE_OPTIONAL_FN(ap_register_include_handler);
cgi_pfn_reg_with_ssi("exec", handle_exec);
static const char * const aszPre[] = { "mod_include.c", NULL };

config

Controls various aspects of the parsing.

<!--#config [timefmt="..."] [sizefmt="..."] [errmsg="..."] -->
<!--#config [echomsg="..."] -->
<!--#config errmsg="[It appears that you don't know how to use SSI]" -->

echomsg

(since 2.1) The value is a message that is sent back to the client if the echo element attempts to echo an undefined variable. This overrides any SSIUndefinedEcho directives.

errmsg

The value is a message that is sent back to the client if an error occurs while parsing the document. This overrides any SSIErrorMsg directives.

sizefmt

The value sets the format to be used which displaying the size of a file. Valid values are bytes for a count in bytes, or abbrev for a count in Kb or Mb as appropriate, for example a size of 1024 bytes will be printed as “1K”.

timefmt

The value is a string to be used by the strftime(3) library routine when printing dates.

echo

Prints one of the include variables. If the variable is unset, the result is determined by the SSIUndefinedEcho directive.

<!--#echo [encoding="none|url|entity"] var="..." [encoding="none|url|entity"] var="..." ... -->

var

The value is the name of the variable to print.

encoding

Specifies how Apache should encode special characters contained in the variable before outputting them. If set to none, no encoding will be done. If set to url, then URL encoding (also known as %-encoding; this is appropriate for use within URLs in links, etc.) will be performed. The default is set to entity, resulting in entity encoding.

exec

Execute external programs

<!--#exec cgi="/cgi-bin/s.cgi" -->
<!--#exec cmd="ls" -->
<!--#include virtual="/cgi-bin/s.cgi?argument=value" -->
<!--#exec cmd="perl /cgi-bin/s.pl args" -->

include

Include a file

<!--#include virtual|file="..." [virtual|file="..."] ... -->

printenv

Print all available variables

<!--#printenv -->

set

Set a value of a variable.

<!--#set var="..." value="..." ... -->
<!--#set var="modified" value="$LAST_MODIFIED" -->
<!--#set var="name" value="AskApache" -->
<!--#set var="date" value="${DATE_LOCAL}_${DATE_GMT}" -->

flastmod

Prints the last modification date of the specified file, subject to the timefmt format specification.

<!--#flastmod virtual|file="..." [virtual|file="..."] ... -->

fsize

Prints the size of the specified file, subject to the sizefmt format specification.

<!--#fsize virtual|file="..." [virtual|file="..."] ... -->

if

The if element works like an if statement in a programming language. The test condition is evaluated and if the result is true, then the text until the next elif, else or endif element is included in the output stream.

<!--#if expr="..." -->
<!--#if expr="${REMOTE_USER} && ${HTTP_USER_AGENT}" -->

<!--#if expr="test_condition" -->
<!--#elif expr="test_condition" -->
<!--#else -->
<!--#endif -->

elif

Used to put text into the output stream if the original test_condition was false.

<!--#elif expr="..." -->

else

Used to put text into the output stream if the original test_condition was false.

<!--#else -->

endif

Ends the if element and is required.

<!--#endif -->

.htaccess directives

1. XBitHack
2. SSIErrorMsg
3. SSITimeFormat
4. SSIStartTag
5. SSIEndTag
6. SSIUndefinedEcho
7. SSIAccessEnable

mod_include Default SSI Values

START_SEQUENCE

– The ending tag for mod_include to recognize and parse as SSI.

ERROR_MSG

[an error occurred while processing this directive] – On Errors parsing SSI.

TIME_FORMAT

%A, %d-%b-%Y %H:%M:%S %Z – Default Time format for DATE

UNDEFINED_ECHO

(none) – When echoing an undefined variable.

SSI Variables

DATE_GMT=Sun Mar  8 22:58:56 2009
DATE_LOCAL=Sun Mar  8 15:58:56 2009
DOCUMENT_NAME=FOOTER.html
DOCUMENT_ROOT=/root-srv/protected/askapache.com/sec
DOCUMENT_URI=/includes/FOOTER.html
GATEWAY_INTERFACE=CGI/1.1
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_ACCEPT_ENCODING=gzip,deflate
HTTP_ACCEPT_LANGUAGE=en-us,en;q=0.5
HTTP_CACHE_CONTROL=max-age=0
HTTP_CONNECTION=keep-alive
HTTP_COOKIE=__qca=12298910-686528-46510;  __utmb=50625.1.0.11311
HTTP_HOST=www.askapache.com
HTTP_KEEP_ALIVE=300
HTTP_REFERER=http://www.askapache.com/htaccess/htaccess.html
HTTP_USER_AGENT=Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)
LAST_MODIFIED=Sun Mar  8 14:53:50 2009
PATH=/bin:/usr/bin:/sbin:/usr/sbin
QUERY_STRING=
REMOTE_ADDR=24.123.215.60
REMOTE_PORT=4785
REQUEST_METHOD=GET
REQUEST_URI=/htaccess/
SCRIPT_FILENAME=/root-srv/protected/askapache.com/sec/includes/FOOTER.html
SCRIPT_NAME=/includes/FOOTER.html
SCRIPT_URI=http://www.askapache.com/htaccess/
SCRIPT_URL=/htaccess/
SERVER_ADDR=64.111.114.111
SERVER_ADMIN=webmaster@askapache.com
SERVER_NAME=www.askapache.com
SERVER_PORT=80
SERVER_PROTOCOL=INCLUDED
SERVER_SIGNATURE=
SERVER_SOFTWARE=Apache/2.0.61 (Unix) PHP/4.4.7 mod_ssl/2.0.63 OpenSSL/0.9.7e mod_fastcgi/2.4.2 DAV/2 SVN/1.4.2
UNIQUE_ID=dnbtH0Bvcm8A2ZHqcAAAAM
USER_NAME=

More SSI Information

1. mod_include
2. Apache Filters
3. Introduction to Server Side Includes
4. Apache Handlers

Advanced Htaccess – SSI, ErrorDocuments, DirectoryIndexing SEO originally appeared on AskApache.com

This is not an introduction to .htaccess… This is the evolution of the best of the best .htaccess on the planet. Its changed over the years with more and better .htaccess tricks using real .htaccess example code. I add my favorite htaccess-related links and sites, results from my htaccess research, and frequently update this article.

You’ve come to the right place if you are looking to acquire mad skills for using .htaccess files.

Originally (2003) this guide was known in certain hacker circles and hidden corners of the net as an ultimate .htaccess due to the powerful htaccess tricks and tips to bypass security on a webhost, and also because many of the tricks and examples were pretty impressive back then in that group.

AskApache Htaccess Journey

Skip this – still under edit

I discovered these tips and tricks mostly while working as a network security penetration specialist hired to find security holes in web hosting environments. Shared hosting is the most common and cheapest form of web-hosting where multiple customers are placed on a single machine and “share” the resources (CPU/RAM/SPACE). The machines are configured to basically ONLY do HTTP and FTP. No shells or any interactive logins, no ssh, just FTP access. That is when I started examining htaccess files in great detail and learned about the incredible untapped power of htaccess. For 99% of the worlds best Apache admins, they don’t use .htaccess much, if AT ALL. It’s much easier, safer, and faster to configure Apache using the httpd.conf file instead. However, this file is almost never readable on shared-hosts, and I’ve never seen it writable. So the only avenue left for those on shared-hosting was and is the .htaccess file, and holy freaking fiber-optics.. it’s almost as powerful as httpd.conf itself!

Most all .htaccess code works in the httpd.conf file, but not all httpd.conf code works in .htaccess files, around 50%. So all the best Apache admins and programmers never used .htaccess files. There was no incentive for those with access to httpd.conf to use htaccess, and the gap grew. It’s common to see “computer gurus” on forums and mailing lists rail against all uses and users of .htaccess files, smugly announcing the well known problems with .htaccess files compared with httpd.conf – I wonder if these “gurus” know the history of the htaccess file, like it’s use in the earliest versions of the HTTP Server- NCSA’s HTTPd, which BTW, became known as Apache HTTP. So you could easily say that htaccess files predates Apache itself.

Once I discovered what .htaccess files could do towards helping me enumerate and exploit security vulnerabilities even on big shared-hosts I focused all my research into .htaccess files, meaning I was reading the venerable Apache HTTP Source code 24/7! I compiled every released version of the Apache Web Server, ever, even NCSA’s, and focused on enumerating the most powerful htaccess directives. Good times! Because my focus was on protocol/file/network vulnerabilites instead of web dev I built up a nice toolbox of htaccess tricks to do unusual things. When I switched over to webdev in 2005 I started using htaccess for websites, not research. I documented most of my favorites and rewrote the htaccess guide for webdevelopers. After some great encouragement on various forums and nets I decided to start a blog to share my work with everyone, AskApache.com was registered, I published my guide, and it was quickly plagiarized and scraped all over the net. Information is freedom, and freedom is information, so this blog has the least restrictive copyright for you. Feel free to modify, copy, republish, sell, or use anything on this site ;)

Htaccess – Evolved

The Hyper Text Transfer Protocol (HTTP) was initiated at the CERN in Geneve (Switzerland), where it emerged (together with the HTML presentation language) from the need to exchange scientific information on a computer network in a simple manner. The first public HTTP implementation only allowed for plain text information, and almost instantaneously became a replacement of the GOPHER service. One of the first text-based browsers was LYNX which still exists today; a graphical HTTP client appeared very quickly with the name NCSA Mosaic. Mosaic was a popular browser back in 1994. Soon the need for a more rich multimedia experience was born, and the markup language provided support for a growing multitude of media types.

Htaccess file know-how will do several things for you:

• Make your website noticeably faster.
• Allow you to debug your server with ease.
• Make your life easier and more rewarding.
• Allow you to work faster and more productively.

What Is .htaccess

Specifically, .htaccess is the default file name of a special configuration file that provides a number of directives (commands) for controlling and configuring the Apache Web Server, and also to control and configure modules that can be built into the Apache installation, or included at run-time like mod_rewrite (for htaccess rewrite), mod_alias (for htaccess redirects), and mod_ssl (for controlling SSL connections).

Htaccess allows for decentralized management of Web Server configurations which makes life very easy for web hosting companies and especially their savvy consumers. They set up and run “server farms” where many hundreds and thousands of web hosting customers are all put on the same Apache Server. This type of hosting is called “virtual hosting” and without .htaccess files would mean that every customer must use the same exact settings as everyone else on their segment. So that is why any half-decent web host allows/enables (DreamHost, Powweb, MediaTemple, GoDaddy) .htaccess files, though few people are aware of it. Let’s just say that if I was a customer on your server-farm, and .htaccess files were enabled, my websites would be a LOT faster than yours, as these configuration files allow you to fully take advantage of and utilize the resources allotted to you by your host. If even 1/10 of the sites on a server-farm took advantage of what they are paying for, the providers would go out of business.

One of the design goals for this server was to maintain external compatibility with the NCSA 1.3 server — that is, to read the same configuration files, to process all the directives therein correctly, and in general to be a drop-in replacement for NCSA. On the other hand, another design goal was to move as much of the server’s functionality into modules which have as little as possible to do with the monolithic server core. The only way to reconcile these goals is to move the handling of most commands from the central server into the modules.

However, just giving the modules command tables is not enough to divorce them completely from the server core. The server has to remember the commands in order to act on them later. That involves maintaining data which is private to the modules, and which can be either per-server, or per-directory. Most things are per-directory, including in particular access control and authorization information, but also information on how to determine file types from suffixes, which can be modified by AddType and DefaultType directives, and so forth. In general, the governing philosophy is that anything which can be made configurable by directory should be; per-server information is generally used in the standard set of modules for information like Aliases and Redirects which come into play before the request is tied to a particular place in the underlying file system.

Another requirement for emulating the NCSA server is being able to handle the per-directory configuration files, generally called .htaccess files, though even in the NCSA server they can contain directives which have nothing at all to do with access control. Accordingly, after URI -> filename translation, but before performing any other phase, the server walks down the directory hierarchy of the underlying filesystem, following the translated pathname, to read any .htaccess files which might be present. The information which is read in then has to be merged with the applicable information from the server’s own config files (either from the sections in access.conf, or from defaults in srm.conf, which actually behaves for most purposes almost exactly like ).

Finally, after having served a request which involved reading .htaccess files, we need to discard the storage allocated for handling them. That is solved the same way it is solved wherever else similar problems come up, by tying those structures to the per-transaction resource pool.

Creating Htaccess Files

Htaccess files use the default filename “.htaccess” but any unix-style file name can be specified from the main server config using the AccessFileName directive. The file isn’t .htaccess.txt, its literally just named .htaccess.

In a Windows Environment like the one I use for work, you can change how Windows opens and views .htaccess files by modifying the Folder Options in explorer. As you can see, on my computer files ending in .htaccess are recognized as having the HTACCESS extension and are handled/opened by Adobe Dreamweaver CS4.

Htaccess Scope

Unlike the main server configuration files like httpd.conf, Htaccess files are read on every request therefore changes in these files take immediate effect. Apache searches all directories and subdirectories that are htaccess-enabled for an .htaccess file which results in performance loss due to file accesses. I’ve never noticed a performance loss but OTOH, I know how to use them. If you do have access to your main server configuration file, you should of course use that instead, and lucky for you ALL the .htaccess tricks and examples can be used there as well (just not vice versa).

Htaccess File Syntax

Htaccess files follow the same syntax as the main Apache configuration files, for powerusers here’s an apache.vim for VI. The one main difference is the context of the directive, which means whether or not that directive is ALLOWED to be used inside of an .htaccess file. Htaccess files are incredibly powerful, and can also be very dangerous as some directives allowed in the main configuration files would allow users/customers to completely bypass security/bandwidth-limits/resource-limits/file-permissions, etc.. About 1/4 of all Apache directives cannot be used inside an .htaccess file (also known as a per-directory context config). The Apache Developers are well-regarded throughout the world as being among some of the best programmers, ever. To enable a disallowed directive inside a .htaccess file would require modifying the source code and re-compiling the server (which they allow and encourage if you are the owner/admin). Here’s a taste of that famous Apache source code that builds the directives allowed in .htaccess file context, the key that tells whether its enabled in .htaccess context is the DIR_CMD_PERMS and then the OR_FILEINFO, which means a directive is enabled dependent on the AllowOverride directive that is only allowed in the main config. First Apache 1.3.0, then Apache 2.2.10

mod_autoindex

AddIcon, add_icon, BY_PATH, DIR_CMD_PERMS, an icon URL followed by one or more filenames
AddIconByType, add_icon, BY_TYPE, DIR_CMD_PERMS, an icon URL followed by one or more MIME types
AddIconByEncoding, add_icon, BY_ENCODING, DIR_CMD_PERMS, an icon URL followed by one or more content encodings
AddAlt, add_alt, BY_PATH, DIR_CMD_PERMS, alternate descriptive text followed by one or more filenames
AddAltByType, add_alt, BY_TYPE, DIR_CMD_PERMS, alternate descriptive text followed by one or more MIME types
AddAltByEncoding, add_alt, BY_ENCODING, DIR_CMD_PERMS, alternate descriptive text followed by one or more content encodings
IndexOptions, add_opts, DIR_CMD_PERMS, RAW_ARGS, one or more index options
IndexIgnore, add_ignore, DIR_CMD_PERMS, ITERATE, one or more file extensions
AddDescription, add_desc, BY_PATH, DIR_CMD_PERMS, Descriptive text followed by one or more filenames
HeaderName, add_header, DIR_CMD_PERMS, TAKE1, a filename
ReadmeName, add_readme, DIR_CMD_PERMS, TAKE1, a filename
FancyIndexing, fancy_indexing, DIR_CMD_PERMS, FLAG, Limited to 'on' or 'off' (superseded by IndexOptions FancyIndexing)
DefaultIcon, ap_set_string_slot, (void *) XtOffsetOf(autoindex_config_rec, default_icon), DIR_CMD_PERMS, TAKE1, an icon URL

mod_rewrite

// mod_rewrite
RewriteEngine, cmd_rewriteengine, OR_FILEINFO, On or Off to enable or disable (default)
RewriteOptions, cmd_rewriteoptions, OR_FILEINFO, List of option strings to set
RewriteBase, cmd_rewritebase, OR_FILEINFO, the base URL of the per-directory context
RewriteCond, cmd_rewritecond, OR_FILEINFO, an input string and a to be applied regexp-pattern
RewriteRule, cmd_rewriterule, OR_FILEINFO, an URL-applied regexp-pattern and a substitution URL
RewriteMap, cmd_rewritemap, RSRC_CONF, a mapname and a filename
RewriteLock, cmd_rewritelock, RSRC_CONF, the filename of a lockfile used for inter-process synchronization
RewriteLog, cmd_rewritelog, RSRC_CONF, the filename of the rewriting logfile
RewriteLogLevel, cmd_rewriteloglevel, RSRC_CONF, the level of the rewriting logfile verbosity (0=none, 1=std, .., 9=max)
RewriteLog, fake_rewritelog, RSRC_CONF, [DISABLED] the filename of the rewriting logfile
RewriteLogLevel, fake_rewritelog, RSRC_CONF, [DISABLED] the level of the rewriting logfile verbosity 

Htaccess Directives

Don’t ask why, but I personally downloaded each major/beta release of the Apache HTTPD source code from version 1.3.0 to version 2.2.10 (all 63 Apache versions!), then I configured and compiled each version for a custom HTTPD installation built from source. This allowed me to find every directive allowed in .htaccess files for each particular version, which has never been done before, or since. YES! I think that is so cool..

An .htaccess directive is basically a command that is specific to a module or builtin to the core that performs a specific task or sets a specific setting for how Apache serves your WebSite. Directives placed in Htaccess files apply to the directory they are in, and all sub-directories. Here’s the 3 top links (official Apache Docs) you will repeatedly use, bookmark/print/save them.

1. Terms Used to Describe Directives
2. Official List of Apache Directives
3. Directive Quick-Reference — with Context

Litespeed .htaccess support

Unlike other lightweight web servers, Apache compatible per-directory configuration overridden is fully supported by LiteSpeed Web Server. With .htacess you can change configurations for any directory under document root on-the-fly, which in most cases is a mandatory feature in shared hosting environment. It is worth noting that enabling .htaccess support in LiteSpeed Web Server will not degrade server’s performance, comparing to Apache’s 40% drop in performance.

Main Server Config Examples

Now lets take a look at some htaccess examples to get a feel for the syntax and some general ideas at the capabilities. Some of the best examples for .htaccess files are included with Apache for main server config files, so lets take a quick look at a couple of them on our way down to the actual .htaccess examples further down the page (this site has thousands, take your time). As you can see, the basic syntax is a line starting with # is a comment, everything else are directives followed by the directive argument.

httpd-multilang-errordoc.conf: The configuration below implements multi-language error documents through content-negotiation

Options IncludesNoExec
AddOutputFilter Includes html
AddHandler type-map var
LanguagePriority en cs de es fr it ja ko nl pl pt-br ro sv tr
ForceLanguagePriority Prefer Fallback
ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var

httpd-manual.conf: Provide local access to the server documentation on your server

SetEnvIf Request_URI ^/manual/(de|en|es|fr|ja|ko|pt-br|ru|tr)/ prefer-language=$1
RedirectMatch 301 ^/manual(?:/(de|en|es|fr|ja|ko|pt-br|ru|tr)){2, }(/.*)?$ /manual/$1$2
LanguagePriority en de es fr ja ko pt-br ru tr
ForceLanguagePriority Prefer Fallback

httpd-languages.conf: Settings for hosting different languages.

DefaultLanguage en
AddLanguage ca .ca
# Just list the languages in decreasing order of preference.
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv tr zh-CN zh-TW
# Commonly used filename extensions to character sets.
AddCharset us-ascii.ascii .us-ascii
AddCharset ISO-8859-1  .iso8859-1  .latin1

httpd-autoindex.conf: Directives controlling the display of server-generated directory listings.

# IndexOptions: Controls the appearance of server-generated directory listings.
IndexOptions FancyIndexing HTMLTable VersionSort
# AddIcon* directives tell the server which icon to show for different files or filename extensions.
AddIconByEncoding (CMP, /icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT, /icons/text.gif) text/*
AddIcon /icons/folder.gif ^^DIRECTORY^^
# DefaultIcon is which icon to show for files which do not have an icon explicitly set.
DefaultIcon /icons/unknown.gif
# AddDescription allows you to place a short description after a file in server-generated indexes.
AddDescription "GZIP compressed document" .gz
# ReadmeName is the name of the README file the server will look for by default, and append to directory listings.
ReadmeName README.html
# HeaderName is the name of a file which should be prepended to directory indexes.
HeaderName HEADER.html

Here are the rest of them if you wanna take a look. (httpd-mpm.conf, httpd-default.conf, httpd-ssl.conf, httpd-info.conf, httpd-vhosts.conf, httpd-dav.conf)

Example .htaccess Files

Here are some samples and examples taken from different .htaccess files I’ve used over the years. Specific solutions are farther down on this page and throughout the site.

# Set the Time Zone of your Server
SetEnv TZ America/Indianapolis
# ServerAdmin:  This address appears on some server-generated pages, such as error documents.
SetEnv SERVER_ADMIN webmaster@askapache.com
# Possible values for the Options directive are "None", "All", or any combination of:
#  Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
Options -ExecCGI -MultiViews -Includes -Indexes FollowSymLinks
# DirectoryIndex: sets the file that Apache will serve if a directory is requested.
DirectoryIndex index.html index.php /index.php
#
# Action lets you define media types that will execute a script whenever
# a matching file is called. This eliminates the need for repeated URL
# pathnames for oft-used CGI file processors.
# Format: Action media/type /cgi-script/location
# Format: Action handler-name /cgi-script/location
#
Action php5-cgi /bin/php.cgi
#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
AddHandler php-cgi .php .inc
# Commonly used filename extensions to character sets.
AddDefaultCharset UTF-8
# AddType allows you to add to or override the MIME configuration
AddType 'application/rdf+xml; charset=UTF-8' .rdf
AddType 'application/xhtml+xml; charset=UTF-8' .xhtml
AddType 'application/xhtml+xml; charset=UTF-8' .xhtml.gz
AddType 'text/html; charset=UTF-8' .html
AddType 'text/html; charset=UTF-8' .html.gz
AddType application/octet-stream .rar .chm .bz2 .tgz .msi .pdf .exe
AddType application/vnd.ms-excel .csv
AddType application/x-httpd-php-source .phps
AddType application/x-pilot .prc .pdb
AddType application/x-shockwave-flash .swf
AddType application/xrds+xml .xrdf
AddType text/plain .ini .sh .bsh .bash .awk .nawk .gawk .csh .var .c .in .h .asc .md5 .sha .sha1
AddType video/x-flv .flv
# AddEncoding allows you to have certain browsers uncompress information on the fly. Note: Not all browsers support this.
AddEncoding x-compress .Z
AddEncoding x-gzip .gz .tgz
# DefaultType: the default MIME type the server will use for a document.
DefaultType text/html
#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off

## MAIN DEFAULTS
Options +ExecCGI -Indexes
DirectoryIndex index.html index.htm index.php
DefaultLanguage en-US
AddDefaultCharset UTF-8
ServerSignature Off
## ENVIRONMENT VARIABLES
SetEnv PHPRC /webroot/includes
SetEnv TZ America/Indianapolis
 
SetEnv SERVER_ADMIN webmaster@askapache.com
## MIME TYPES
AddType video/x-flv .flv
AddType application/x-shockwave-flash .swf
AddType image/x-icon .ico
## FORCE FILE TO DOWNLOAD INSTEAD OF APPEAR IN BROWSER
# http://www.htaccesselite.com/addtype-addhandler-action-vf6.html
AddType application/octet-stream .mov .mp3 .zip
## ERRORDOCUMENTS
# http://askapache.com/htaccess/apache-status-code-headers-errordocument.html
ErrorDocument 400 /e400/
ErrorDocument 401 /e401/
ErrorDocument 402 /e402/
ErrorDocument 403 /e403/
ErrorDocument 404 /e404/
#
# Handlers be builtin, included in a module, or added with Action directive
# default-handler: default, handles static content (core)
#   send-as-is: Send file with HTTP headers (mod_asis)
#   cgi-script: treat file as CGI script (mod_cgi)
#    imap-file: Parse as an imagemap rule file (mod_imap)
#   server-info: Get server config info (mod_info)
#  server-status: Get server status report (mod_status)
#    type-map: type map file for content negotiation (mod_negotiation)
#  fastcgi-script: treat file as fastcgi script (mod_fastcgi)
#
# http://www.askapache.com/php/custom-phpini-tips-and-tricks.html
## PARSE AS CGI
AddHandler cgi-script .cgi .pl .spl
## RUN PHP AS APACHE MODULE
AddHandler application/x-httpd-php .php .htm
## RUN PHP AS CGI
AddHandler php-cgi .php .htm
## CGI PHP WRAPPER FOR CUSTOM PHP.INI
AddHandler phpini-cgi .php .htm
Action phpini-cgi /cgi-bin/php5-custom-ini.cgi
## FAST-CGI SETUP WITH PHP-CGI WRAPPER FOR CUSTOM PHP.INI
AddHandler fastcgi-script .fcgi
AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php5-wrapper.fcgi
## CUSTOM PHP CGI BINARY SETUP
AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php.cgi
## PROCESS SPECIFIC FILETYPES WITH CGI-SCRIPT
Action image/gif /cgi-bin/img-create.cgi
## CREATE CUSTOM HANDLER FOR SPECIFIC FILE EXTENSIONS
AddHandler custom-processor .ssp
Action custom-processor /cgi-bin/myprocessor.cgi
### HEADER CACHING
# http://www.askapache.com/htaccess/speed-up-sites-with-htaccess-caching.html
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico)$">
Header set Cache-Control "max-age=2592000"
</FilesMatch>
<FilesMatch "\.(js|css|pdf|swf)$">
Header set Cache-Control "max-age=604800"
</FilesMatch>
<FilesMatch "\.(html|htm|txt)$">
Header set Cache-Control "max-age=600"
</FilesMatch>
<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
Header unset Cache-Control
</FilesMatch>
## ALTERNATE EXPIRES CACHING
# htaccesselite.com/d/use-htaccess-to-speed-up-your-site-discussion-vt67.html
ExpiresActive On
ExpiresDefault A604800
ExpiresByType image/x-icon A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType text/css A2592000
ExpiresByType text/html A300
<FilesMatch "\.(pl|php|cgi|spl|scgi|fcgi)$">
ExpiresActive Off
</FilesMatch>
## META HTTP-EQUIV REPLACEMENTS
<FilesMatch "\.(html|htm|php)$">
Header set imagetoolbar "no"
</FilesMatch>

Here are some default MOD_REWRITE code examples.

## REWRITE DEFAULTS
RewriteEngine On
RewriteBase /
## REQUIRE SUBDOMAIN
RewriteCond %{HTTP_HOST} !^$
RewriteCond %{HTTP_HOST} !^subdomain\.askapache\.com$ [NC]
RewriteRule ^/(.*)$ http://subdomain.askapache.com/$1 [L,R=301]
## SEO REWRITES
RewriteRule ^(.*)/ve/(.*)$ $1/voluntary-employee/$2 [L,R=301]
RewriteRule ^(.*)/hsa/(.*)$ $1/health-saving-account/$2 [L,R=301]
## WORDPRESS
RewriteCond %{REQUEST_FILENAME} !-f  # Existing File
RewriteCond %{REQUEST_FILENAME} !-d  # Existing Directory
RewriteRule . /index.php [L]
## ALTERNATIVE ANTI-HOTLINKING
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(subdomain\.)?askapache.com/.*$ [NC]
RewriteRule ^.*\.(bmp|tif|gif|jpg|jpeg|jpe|png)$ - [F]
## REDIRECT HOTLINKERS
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(subdomain\.)?askapache.com/.*$ [NC]
RewriteRule ^.*\.(bmp|tif|gif|jpg|jpeg|jpe|png)$ http://google.com [R]
## DENY REQUEST BASED ON REQUEST METHOD
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK|OPTIONS|HEAD)$ [NC]
RewriteRule ^.*$ - [F]
## REDIRECT UPLOADS
RewriteCond %{REQUEST_METHOD} ^(PUT|POST)$ [NC]
RewriteRule ^(.*)$ /cgi-bin/form-upload-processor.cgi?p=$1 [L,QSA]
## REQUIRE SSL EVEN WHEN MOD_SSL IS NOT LOADED
RewriteCond %{HTTPS} !=on [NC]
RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L]
### ALTERNATATIVE TO USING ERRORDOCUMENT
# http://www.htaccesselite.com/d/htaccess-errordocument-examples-vt11.html
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^.*$ /error.php [L]
## SEO REDIRECTS
Redirect 301 /2006/oldfile.html http://subdomain.askapache.com/newfile.html
RedirectMatch 301 /o/(.*)$ http://subdomain.askapache.com/s/dl/$1

Examples of protecting your files and securing with password protection.

#
# Require (user|group|valid-user) (username|groupname)
#
## BASIC PASSWORD PROTECTION
AuthType basic
AuthName "prompt"
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user
## ALLOW FROM IP OR VALID PASSWORD
Require valid-user
Allow from 192.168.1.23
Satisfy Any
## PROTECT FILES
<FilesMatch "\.(htaccess|htpasswd|ini|phps|fla|psd|log|sh)$">
Order Allow,Deny
Deny from all
</FilesMatch>
## PREVENT HOTLINKING
SetEnvIfNoCase Referer "^http://subdomain.askapache.com/" good
SetEnvIfNoCase Referer "^$" good
<FilesMatch "\.(png|jpg|jpeg|gif|bmp|swf|flv)$">
Order Deny,Allow
Deny from all
Allow from env=good
ErrorDocument 403 http://www.google.com/intl/en_ALL/images/logo.gif
ErrorDocument 403 /images/you_bad_hotlinker.gif
</FilesMatch>
## LIMIT UPLOAD FILE SIZE TO PROTECT AGAINST DOS ATTACK
#bytes, 0-2147483647(2GB)
LimitRequestBody 10240000
## MOST SECURE WAY TO REQUIRE SSL
# http://www.askapache.com/htaccess/apache-ssl-in-htaccess-examples.html
SSLOptions +StrictRequire
SSLRequireSSL
SSLRequire %{HTTP_HOST} eq "askapache.com"
ErrorDocument 403 https://askapache.com
## COMBINED DEVELOPER HTACCESS CODE-USE THIS
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico|js|css|pdf|swf|html|htm|txt)$">
Header set Cache-Control "max-age=5"
</FilesMatch>
AuthType basic
AuthName "Ooops! Temporarily Under Construction..."
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user      # password prompt for everyone else
Order Deny,Allow
Deny from all
Allow from 192.168.64.5   # Your, the developers IP address
Allow from w3.org      # css/xhtml check jigsaw.w3.org/css-validator/
Allow from googlebot.com   # Allows google to crawl your pages
Satisfy Any        # no password required if host/ip is Allowed
## DONT HAVE TO EMPTY CACHE OR RELOAD TO SEE CHANGES
ExpiresDefault A5 #If using mod_expires
<FilesMatch "\.(flv|gif|jpg|jpeg|png|ico|js|css|pdf|swf|html|htm|txt)$">
Header set Cache-Control "max-age=5"
</FilesMatch>
## ALLOW ACCESS WITH PASSWORD OR NO PASSWORD FOR SPECIFIC IP/HOSTS
AuthType basic
AuthName "Ooops! Temporarily Under Construction..."
AuthUserFile /.htpasswd
AuthGroupFile /dev/null
Require valid-user      # password prompt for everyone else
Order Deny,Allow
Deny from all
Allow from 192.168.64.5   # Your, the developers IP address
Allow from w3.org      # css/xhtml check jigsaw.w3.org/css-validator/
Allow from googlebot.com   # Allows google to crawl your pages
Satisfy Any        # no password required if host/ip is Allowed

Example .htaccess Code Snippets

Here are some specific examples, this is the most popular section of this page. Updated frequently.

Redirect Everyone Except IP address to alternate page

ErrorDocument 403 http://www.yahoo.com/
Order deny,allow
Deny from all
Allow from 208.113.134.190

When developing sites

This lets google crawl the page, lets me access without a password, and lets my client access the page WITH a password. It also allows for XHTML and CSS validation! (w3.org)

AuthName "Under Development"
AuthUserFile /home/sitename.com/.htpasswd
AuthType basic
Require valid-user
Order deny,allow
Deny from all
Allow from 208.113.134.190 w3.org htmlhelp.com googlebot.com
Satisfy Any

Fix double-login prompt

Redirect non-https requests to https server and ensure that .htpasswd authorization can only be entered across HTTPS

SSLOptions +StrictRequire
SSLRequireSSL
SSLRequire %{HTTP_HOST} eq "askapache.com"
ErrorDocument 403 https://askapache.com

Set Timezone of the Server (GMT)

SetEnv TZ America/Indianapolis

Administrator Email for ErrorDocument

SetEnv SERVER_ADMIN webmaster@google.com

ServerSignature for ErrorDocument

ServerSignature off | on | email

Charset and Language headers

Article: Setting Charset in htaccess, and article by Richard Ishida

AddDefaultCharset UTF-8
DefaultLanguage en-US

Disallow Script Execution

Options -ExecCGI
AddHandler cgi-script .php .pl .py .jsp .asp .htm .shtml .sh .cgi

Deny Request Methods

RewriteCond %{REQUEST_METHOD} !^(GET|HEAD|OPTIONS|POST|PUT)
RewriteRule .* - [F]

Force “File Save As” Prompt

AddType application/octet-stream .avi .mpg .mov .pdf .xls .mp4

Show CGI Source Code

RemoveHandler cgi-script .pl .py .cgi
AddType text/plain .pl .py .cgi

Serve all .pdf files on your site using .htaccess and mod_rewrite with the php script.

RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^(.+)\.pdf$  /cgi-bin/pdf.php?file=$1 [L,NC,QSA]

Rewrite to www

RewriteCond %{REQUEST_URI} !^/(robots\.txt|favicon\.ico|sitemap\.xml)$
RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Rewrite to www dynamically

RewriteCond %{REQUEST_URI} !^/robots\.txt$ [NC]
RewriteCond %{HTTP_HOST} !^www\.[a-z-]+\.[a-z]{2,6} [NC]
RewriteCond %{HTTP_HOST} ([a-z-]+\.[a-z]{2,6})$   [NC]
RewriteRule ^/(.*)$ http://%1/$1 [R=301,L]

301 Redirect Old File

Redirect 301 /old/file.html http://www.askapache.com/new/file.html

301 Redirect Entire Directory

RedirectMatch 301 /blog(.*) http://www.askapache.com/$1

Protecting your php.cgi

<FilesMatch "^php5?\.(ini|cgi)$">
Order Deny,Allow
Deny from All
Allow from env=REDIRECT_STATUS
</FilesMatch>

Set Cookie based on Request

This code sends the Set-Cookie header to create a cookie on the client with the value of a matching item in 2nd parantheses.

RewriteEngine On
RewriteBase /
RewriteRule ^(.*)(de|es|fr|it|ja|ru|en)/$ - [co=lang:$2:.askapache.com:7200:/]

Set Cookie with env variable

Header set Set-Cookie "language=%{lang}e; path=/;" env=lang

Custom ErrorDocuments

ErrorDocument 100 /100_CONTINUE
ErrorDocument 101 /101_SWITCHING_PROTOCOLS
ErrorDocument 102 /102_PROCESSING
ErrorDocument 200 /200_OK
ErrorDocument 201 /201_CREATED
ErrorDocument 202 /202_ACCEPTED
ErrorDocument 203 /203_NON_AUTHORITATIVE
ErrorDocument 204 /204_NO_CONTENT
ErrorDocument 205 /205_RESET_CONTENT
ErrorDocument 206 /206_PARTIAL_CONTENT
ErrorDocument 207 /207_MULTI_STATUS
ErrorDocument 300 /300_MULTIPLE_CHOICES
ErrorDocument 301 /301_MOVED_PERMANENTLY
ErrorDocument 302 /302_MOVED_TEMPORARILY
ErrorDocument 303 /303_SEE_OTHER
ErrorDocument 304 /304_NOT_MODIFIED
ErrorDocument 305 /305_USE_PROXY
ErrorDocument 307 /307_TEMPORARY_REDIRECT
ErrorDocument 400 /400_BAD_REQUEST
ErrorDocument 401 /401_UNAUTHORIZED
ErrorDocument 402 /402_PAYMENT_REQUIRED
ErrorDocument 403 /403_FORBIDDEN
ErrorDocument 404 /404_NOT_FOUND
 
ErrorDocument 405 /405_METHOD_NOT_ALLOWED
ErrorDocument 406 /406_NOT_ACCEPTABLE
ErrorDocument 407 /407_PROXY_AUTHENTICATION_REQUIRED
ErrorDocument 408 /408_REQUEST_TIME_OUT
ErrorDocument 409 /409_CONFLICT
ErrorDocument 410 /410_GONE
ErrorDocument 411 /411_LENGTH_REQUIRED
ErrorDocument 412 /412_PRECONDITION_FAILED
ErrorDocument 413 /413_REQUEST_ENTITY_TOO_LARGE
ErrorDocument 414 /414_REQUEST_URI_TOO_LARGE
ErrorDocument 415 /415_UNSUPPORTED_MEDIA_TYPE
ErrorDocument 416 /416_RANGE_NOT_SATISFIABLE
ErrorDocument 417 /417_EXPECTATION_FAILED
ErrorDocument 422 /422_UNPROCESSABLE_ENTITY
ErrorDocument 423 /423_LOCKED
ErrorDocument 424 /424_FAILED_DEPENDENCY
ErrorDocument 426 /426_UPGRADE_REQUIRED
ErrorDocument 500 /500_INTERNAL_SERVER_ERROR
ErrorDocument 501 /501_NOT_IMPLEMENTED
ErrorDocument 502 /502_BAD_GATEWAY
ErrorDocument 503 /503_SERVICE_UNAVAILABLE
ErrorDocument 504 /504_GATEWAY_TIME_OUT
ErrorDocument 505 /505_VERSION_NOT_SUPPORTED
ErrorDocument 506 /506_VARIANT_ALSO_VARIES
ErrorDocument 507 /507_INSUFFICIENT_STORAGE
ErrorDocument 510 /510_NOT_EXTENDED

Implementing a Caching Scheme with .htaccess

# year
<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|swf|mp3|mp4)$">
Header set Cache-Control "public"
Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
Header unset Last-Modified
</FilesMatch>
#2 hours
<FilesMatch "\.(html|htm|xml|txt|xsl)$">
Header set Cache-Control "max-age=7200, must-revalidate"
</FilesMatch>
<FilesMatch "\.(js|css)$">
SetOutputFilter DEFLATE
Header set Expires "Thu, 15 Apr 2010 20:00:00 GMT"
</FilesMatch>

Password Protect single file

<Files login.php>
AuthName "Prompt"
AuthType Basic
AuthUserFile /home/askapache.com/.htpasswd
Require valid-user
</Files>

Password Protect multiple files

<FilesMatch "^(private|phpinfo)\.*$">
AuthName "Development"
AuthUserFile /.htpasswd
AuthType basic
Require valid-user
</FilesMatch>

Send Custom Headers

Header set P3P "policyref=\"http://www.askapache.com/w3c/p3p.xml\""
Header set X-Pingback "http://www.askapache.com/xmlrpc.php"
Header set Content-Language "en-US"
Header set Vary "Accept-Encoding"

Blocking based on User-Agent Header

SetEnvIfNoCase ^User-Agent$ .*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures) HTTP_SAFE_BADBOT
SetEnvIfNoCase ^User-Agent$ .*(libwww-perl|aesop_com_spiderman) HTTP_SAFE_BADBOT
Deny from env=HTTP_SAFE_BADBOT

Blocking with RewriteCond

RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
RewriteRule . - [F,L]

.htaccess for mod_php

SetEnv PHPRC /location/todir/containing/phpinifile

.htaccess for php as cgi

AddHandler php-cgi .php .htm
Action php-cgi /cgi-bin/php5.cgi

Shell wrapper for custom php.ini

#!/bin/sh
export PHP_FCGI_CHILDREN=3
exec php5.cgi -c /abs/php5/php.ini

Add values from HTTP Headers

SetEnvIfNoCase ^If-Modified-Since$ "(.+)" HTTP_IF_MODIFIED_SINCE=$1
SetEnvIfNoCase ^If-None-Match$ "(.+)" HTTP_IF_NONE_MATCH=$1
SetEnvIfNoCase ^Cache-Control$ "(.+)" HTTP_CACHE_CONTROL=$1
SetEnvIfNoCase ^Connection$ "(.+)" HTTP_CONNECTION=$1
SetEnvIfNoCase ^Keep-Alive$ "(.+)" HTTP_KEEP_ALIVE=$1
SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1
SetEnvIfNoCase ^Cookie$ "(.+)" HTTP_MY_COOKIE=$1

Stop hotlinking

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?askapache.com/.*$ [NC]
RewriteRule \.(gif|jpg|swf|flv|png)$ http://www.askapache.com/feed.gif [R=302,L]

Turn logging off for IP

SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "nolog,noauditlog,pass"

Turn logging on for IP

SecFilterSelective REMOTE_ADDR "!^208\.113\.183\.103" "nolog,noauditlog,pass"
SecFilterSelective REMOTE_ADDR "208\.113\.183\.103" "log,auditlog,pass"

Advanced Mod_Rewrite Examples

Here are some specific examples taken mostly from my WordPress Password Protection plugin, that also does alot more than password protection as you can see from the following mod_rewrite examples. These are a few of the mod_rewrite uses that BlogSecurity declared pushed the boundaries of Mod_Rewrite! Some of these snippets are quite exotic and unlike anything you may have seen before, also only for those who understand them as they can kill a website pretty quick.

Directory Protection

Enable the DirectoryIndex Protection, preventing directory index listings and defaulting. [Disable]

Options -Indexes
DirectoryIndex index.html index.php /index.php

Password Protect wp-login.php

Requires a valid user/pass to access the login page[401]

<Files wp-login.php>
Order Deny,Allow
Deny from All
Satisfy Any
AuthName "Protected By AskApache"
AuthUserFile /home/askapache.com/.htpasswda1
AuthType Basic
Require valid-user
</Files>

Password Protect wp-admin

Requires a valid user/pass to access any non-static (css, js, images) file in this directory.[401]

Options -ExecCGI -Indexes +FollowSymLinks -Includes
DirectoryIndex index.php /index.php
Order Deny,Allow
Deny from All
Satisfy Any
AuthName "Protected By AskApache"
AuthUserFile /home/askapache.com/.htpasswda1
AuthType Basic
Require valid-user
<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|mp3|mpg|mp4|mov|wav|wmv|png|gif|swf|css|js)$">
Allow from All
</FilesMatch>
<FilesMatch "(async-upload)\.php$">
<IfModule mod_security.c>
SecFilterEngine Off
</IfModule>
Allow from All
</FilesMatch>

Protect wp-content

Denies any Direct request for files ending in .php with a 403 Forbidden.. May break plugins/themes [401]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /wp-content/.*$ [NC]
RewriteCond %{REQUEST_FILENAME} !^.+flexible-upload-wp25js.php$
RewriteCond %{REQUEST_FILENAME} ^.+\.(php|html|htm|txt)$
RewriteRule .* - [F,NS,L]

Protect wp-includes

Denies any Direct request for files ending in .php with a 403 Forbidden.. May break plugins/themes [403]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /wp-includes/.*$ [NC]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ /wp-includes/js/.+/.+\ HTTP/ [NC]
RewriteCond %{REQUEST_FILENAME} ^.+\.php$
RewriteRule .* - [F,NS,L]

Common Exploits

Block common exploit requests with 403 Forbidden. These can help alot, may break some plugins. [403]

RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ ///.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\=?(http|ftp|ssl|https):/.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\?\?.*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\.(asp|ini|dll).*\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\.(htpasswd|htaccess|aahtpasswd).*\ HTTP/ [NC]
RewriteRule .* - [F,NS,L]

Stop Hotlinking

Denies any request for static files (images, css, etc) if referrer is not local site or empty. [403]

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteCond %{HTTP_REFERER} !^http://www.askapache.com.*$ [NC]
RewriteRule \.(ico|pdf|flv|jpg|jpeg|mp3|mpg|mp4|mov|wav|wmv|png|gif|swf|css|js)$ - [F,NS,L]

Safe Request Methods

Denies any request not using GET,PROPFIND,POST,OPTIONS,PUT,HEAD[403]

RewriteCond %{REQUEST_METHOD} !^(GET|HEAD|POST|PROPFIND|OPTIONS|PUT)$ [NC]
RewriteRule .* - [F,NS,L]

Forbid Proxies

Denies any POST Request using a Proxy Server. Can still access site, but not comment. See Perishable Press [403]

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP:VIA}%{HTTP:FORWARDED}%{HTTP:USERAGENT_VIA}%{HTTP:X_FORWARDED_FOR}%{HTTP:PROXY_CONNECTION} !^$ [OR]
RewriteCond %{HTTP:XPROXY_CONNECTION}%{HTTP:HTTP_PC_REMOTE_ADDR}%{HTTP:HTTP_CLIENT_IP} !^$
RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteRule .* - [F,NS,L]

Real wp-comments-post.php

Denies any POST attempt made to a non-existing wp-comments-post.php[403]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*/wp-comments-post\.php.*\ HTTP/ [NC]
RewriteRule .* - [F,NS,L]

HTTP PROTOCOL

Denies any badly formed HTTP PROTOCOL in the request, 0.9, 1.0, and 1.1 only[403]

RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ .+\ HTTP/(0\.9|1\.0|1\.1) [NC]
RewriteRule .* - [F,NS,L]

SPECIFY CHARACTERS

Denies any request for a url containing characters other than “a-zA-Z0-9.+/-?=&” – REALLY helps but may break your site depending on your links. [403]

RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteCond %{THE_REQUEST} !^[A-Z]{3,9}\ [a-zA-Z0-9\.\+_/\-\?\=\&]+\ HTTP/ [NC]
RewriteRule .* - [F,NS,L]

BAD Content Length

Denies any POST request that doesnt have a Content-Length Header[403]

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP:Content-Length} ^$
RewriteCond %{REQUEST_URI} !^/(wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteRule .* - [F,NS,L]

BAD Content Type

Denies any POST request with a content type other than application/x-www-form-urlencoded|multipart/form-data[403]

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP:Content-Type} !^(application/x-www-form-urlencoded|multipart/form-data.*(boundary.*)?)$ [NC]
RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteRule .* - [F,NS,L]

Missing HTTP_HOST

Denies requests that dont contain a HTTP HOST Header.[403]

RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteCond %{HTTP_HOST} ^$
RewriteRule .* - [F,NS,L]

Bogus Graphics Exploit

Denies obvious exploit using bogus graphics[403]

RewriteCond %{HTTP:Content-Disposition} \.php [NC]
RewriteCond %{HTTP:Content-Type} image/.+ [NC]
RewriteRule .* - [F,NS,L]

No UserAgent, Not POST

Denies POST requests by blank user-agents. May prevent a small number of visitors from POSTING. [403]

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REQUEST_URI} !^/(wp-login.php|wp-admin/|wp-content/plugins/|wp-includes/).* [NC]
RewriteRule .* - [F,NS,L]

No Referer, No Comment

Denies any comment attempt with a blank HTTP_REFERER field, highly indicative of spam. May prevent some visitors from POSTING. [403]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*/wp-comments-post\.php.*\ HTTP/ [NC]
RewriteCond %{HTTP_REFERER} ^-?$
RewriteRule .* - [F,NS,L]

Trackback Spam

Denies obvious trackback spam. See Holy Shmoly! [403]

RewriteCond %{REQUEST_METHOD} =POST
RewriteCond %{HTTP_USER_AGENT} ^.*(opera|mozilla|firefox|msie|safari).*$ [NC]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.+/trackback/?\ HTTP/ [NC]
RewriteRule .* - [F,NS,L]

Map all URIs except those corresponding to existing files to a handler

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-d
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
RewriteRule . /script.php

Map any request to a handler

In the case where all URIs should be sent to the same place (including potentially requests for static content) the method to use depends on the type of the handler. For php scripts, use:
For other handlers such as php scripts, use:

RewriteEngine On
RewriteCond %{REQUEST_URI} !=/script.php
RewriteRule .* /script.php

And for CGI scripts:

ScriptAliasMatch .* /var/www/script.cgi

Map URIs corresponding to existing files to a handler instead

RewriteEngine On
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -d [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} -f
RewriteCond %{REQUEST_URI} !=/script.php
RewriteRule .* /script.php

If the existing files you wish to have handled by your script have a common set of file extensions distinct from that of the hander, you can bypass mod_rewrite and use instead mod_actions. Let’s say you want all .html and .tpl files to be dealt with by your script:

Action foo-action /script.php
AddHandler foo-action html tpl

Deny access if var=val contains the string foo.

RewriteCond %{QUERY_STRING} foo
RewriteRule ^/url - [F]

Removing the Query String

RewriteRule ^/url /url?

Adding to the Query String

Keep the existing query string using the Query String Append flag, but add var=val to the end.

RewriteRule ^/url /url?var=val [QSA]

Rewriting For Certain Query Strings

Rewrite URLs like http://askapache.com/url1?var=val to http://askapache.com/url2?var=val but don’t rewrite if val isn’t present.

RewriteCond %{QUERY_STRING} val
RewriteRule ^/url1 /url2

Modifying the Query String

Change any single instance of val in the query string to other_val when accessing /path. Note that %1 and %2 are back-references to the matched part of the regular expression in the previous RewriteCond.

RewriteCond %{QUERY_STRING} ^(.*)val(.*)$
RewriteRule /path /path?%1other_val%2

Best .htaccess Articles

.htaccess for Webmasters

• Process certain requests for files using a cgi script
• Process Requests with certain Request Methods
• Make any file be a certain filetype
• Use IfModule directive for robust code

Mod_Rewrite URL Rewriting

Undocumented techniques and methods will allow you to utilize mod_rewrite at an “expert level” by showing you how to unlock its secrets.

• Check for a key in QUERY_STRING
• Block access to files during certain hours of the day
• Rewrite underscores to hyphens for SEO URL
• Redirecting WordPress Feeds to Feedburner

301 Redirects without mod_rewrite

• Redirect single url
• Redirect to new Domain

Secure PHP with .htaccess

If you have a php.cgi or php.ini file in your /cgi-bin/ directory or other pub directory, try requesting them from your web browser. If your php.ini shows up or worse you are able to execute your php cgi, you’ll need to secure it ASAP. This shows several ways to secure these files, and other interpreters like perl, fastCGI, bash, csh, etc.

.htaccess Cookie Manipulation

Fresh .htaccess code for you! Check out the Cookie Manipulation and environment variable usage with mod_rewrite! I also included a couple Mod_Security .htaccess examples. Enjoy!

• Mod_Rewrite .htaccess Examples
• Cookie Manipulation and Tests with mod_rewrite
• Setting Environment Variables
• Using the Environment Variable
• Mod_Security .htaccess Examples

.htaccess Caching

• Speed Up Sites with htaccess Caching
• htaccess time cheat sheet

Password Protection and Authentication

• Require password for single file
• Example .htaccess file for password protection

Control HTTP Headers

• Prevent Caching 100%
• Remove IE imagetoolbar without meta tag
• Add Privacy (P3P) Header to your site
• Add language and charset headers without meta tags

Blocking Spam and bad Bots

Want to block a bad robot or web scraper using .htaccess files? Here are 2 methods that illustrate blocking 436 various user-agents. You can block them using either SetEnvIf methods, or by using Rewrite Blocks.

PHP htaccess tips

By using some cool .htaccess tricks we can control PHP to be run as a cgi or a module. If php is run as a cgi then we need to compile it ourselves or use .htaccess to force php to use a local php.ini file. If it is running as a module then we can use various directives supplied by that modules in .htaccess

• When php run as CGI
• Use a custom php.ini with mod_php or php as a cgi
• When php run as Apache Module (mod_php)
• When cgi php is run with wrapper (FastCGI)

HTTP to HTTPS Redirects with mod_rewrite

This is freaking sweet if you use SSL I promise you! Basically instead of having to check for HTTPS using a RewriteCond %{HTTPS} =on for every redirect that can be either HTTP or HTTPS, I set an environment variable once with the value “http” or “https” if HTTP or HTTPS is being used for that request, and use that env variable in the RewriteRule.

SSL in .htaccess

• Redirect non-https requests to https server
• Rewrite non-https to HTTPS without mod_ssl!
• Redirect everything served on port 80 to HTTPS URI

SetEnvIf and SetEnvIfNoCase in .htaccess

• Unique mod_setenvif Variables
• Populates HTTP_MY_ Variables with mod_setenvif variable values
• Allows only if HOST Header is present in request
• Add values from HTTP Headers

Site Security with .htaccess

chmod .htpasswd files 640, chmod .htaccess 644, php files 600, and chmod files that you really dont want people to see as 400. (NEVER chmod 777, try 766)

• CHMOD your files
• Prevent access to .htaccess and .htpasswd files
• Show Source Code instead of executing
• Securing directories: Remove ability to execute scripts
• .htaccess ErrorDocuments

.htaccess Security with MOD_SECURITY

• mod_security + mod_rewrite
• Block Spam by examining POST form fields
• Disabling mod_security conditionally per IP
• Disabling mod_security with .htaccess Authorization
• Block WordPress Spam Forever!
• Force Any Connections to be Paused a set number of ms
• ModSecurity Debugging and Logging
• Turn Off/On Logging JUST for your IP Address
• Mod_Security Directives for DreamHost
• Example httpd.conf mod_security rule files

Merging Notes

The order of merging is:

1. <Directory> (except regular expressions) and .htaccess done simultaneously (with .htaccess, if allowed, overriding <Directory>)
2. <DirectoryMatch> (and <Directory ~>)
3. <Files> and <FilesMatch> done simultaneously
4. <Location> and <LocationMatch> done simultaneously

Below is an artificial example to show the order of merging. Assuming they all apply to the request, the directives in this example will be applied in the order:

A > B > C > D > E

.

<Location />
E
</Location>
<Files askapache.txt>
D
</Files>
<VirtualHost *>
<Directory /a/b>
B
</Directory>
</VirtualHost>
<DirectoryMatch "^.*b$">
C
</DirectoryMatch>
<Directory /a/b>
A
</Directory>

My Favorite .htaccess Links

These are just some of my favorite .htaccess resources. I’m really into doing your own hacking to get knowledge and these links are all great resources in that respect. I’m really interested in new or unusual htaccess solutions or htaccess hacks using .htaccess files, so let me know if you find one.

NCSA HTTPd Tutorials

Robert Hansen
Here’s a great Hardening HTAccess part 1, part 2, part 3 article that goes into detail about some of the rarer security applications for .htaccess files.

SAMAXES
Some very detailed and helpful .htaccess articles, such as the “.htaccess – gzip and cache your site for faster loading and bandwidth saving.”

PerishablePress
Stupid .htaccess tricks is probably the best explanation online for many of the best .htaccess solutions, including many from this page. Unlike me they are fantastic writers, even for technical stuff they are very readable, so its a good blog to kick back on and read. They also have a fantastic article detailing how to block/deny specific requests using mod_rewrite.

BlogSecurity
Mostly a site for… blog security (which is really any web-app security) this blog has a few really impressive articles full of solid information for Hardening WordPress with .htaccess among more advanced topics that can be challenging but effective. This is a good site to subscribe to their feed, they publish plugin exploits and wordpress core vulnerabilities quite a bit.

Check-These
Oldschool security/unix dude with some incredibly detailed mod_rewrite tutorials, helped me the most when I first got into this, and a great guy too. See: Basic Mod_Rewrite Guide, and Advanced Mod_Rewrite Tutorial

Reaper-X
Alot of .htaccess tutorials and code. See: Hardening WordPress with Mod Rewrite and htaccess

jdMorgan
jdMorgan is the Moderator of the Apache Forum at WebmasterWorld, a great place for answers. In my experience he can answer any tough question pertaining to advanced .htaccess usage, haven’t seen him stumped yet.

The W3C
Setting Charset in .htaccess is very informative.

Holy Shmoly!
A great blogger with analysis of attacks and spam. See: More ways to stop spammers and unwanted traffic.

Apache Week
A partnership with Red Hat back in the 90′s that produced some excellent documentation.

Corz
Here’s a resource that I consider to have some of the most creative and ingenious ideas for .htaccess files, although the author is somewhat of a character ;) Its a trip trying to navigate around the site, a fun trip. Its like nothing I’ve ever seen. There are only a few articles on the site, but the htaccess articles are very original and well-worth a look. See: htaccess tricks and tips.

Htaccess Directives

This is an AskApache.com exclusive you won’t find this anywhere else.

Directory, DirectoryMatch, Files, FilesMatch, IfDefine, IfVersion, IfModule, Limit, LimitExcept, Location, LocationMatch, Proxy, ProxyMatch, VirtualHost, AcceptMutex, AcceptPathInfo, AccessFileName, Action, AddCharset, AddDefaultCharset, AddDescription, AddEncoding, AddHandler, AddInputFilter, AddLanguage, AddOutputFilter, AddOutputFilterByType, AddType, Alias, AliasMatch, AllowCONNECT, AllowOverride, Anonymous, Anonymous_Authoritative, Anonymous_LogEmail, Anonymous_MustGiveEmail, Anonymous_NoUserId, Anonymous_VerifyEmail, AuthAuthoritative, AuthDBMAuthoritative, AuthDBMGroupFile, AuthDBMType, AuthDBMUserFile, AuthDigestAlgorithm, AuthDigestDomain, AuthDigestFile, AuthDigestGroupFile, AuthDigestNcCheck, AuthDigestNonceFormat, AuthDigestNonceLifetime, AuthDigestQop, AuthDigestShmemSize, AuthGroupFile, AuthName, AuthType, AuthUserFile, BS2000Account, BrowserMatch, BrowserMatchNoCase, CacheNegotiatedDocs, CharsetDefault, CharsetOptions, CharsetSourceEnc, CheckSpelling, ContentDigest, CookieDomain, CookieExpires, CookieName, CookieStyle, CookieTracking, CoreDumpDirectory, DAV, DAVDepthInfinity, DAVMinTimeout, DefaultIcon, DefaultLanguage, DefaultType, DocumentRoot, ErrorDocument, ErrorLog, ExtFilterDefine, ExtFilterOptions, FancyIndexing, FileETag, ForceLanguagePriority, ForceType, GprofDir, Header, HeaderName, HostnameLookups, IdentityCheck, ImapBase, ImapDefault, ImapMenu, Include, IndexIgnore, LanguagePriority, LimitRequestBody, LimitRequestFields, LimitRequestFieldsize, LimitRequestLine, LimitXMLRequestBody, LockFile, LogLevel, MaxRequestsPerChild, MultiviewsMatch, NameVirtualHost, NoProxy, Options, PassEnv, PidFile, Port, ProxyBlock, ProxyDomain, ProxyErrorOverride, ProxyIOBufferSize, ProxyMaxForwards, ProxyPass, ProxyPassReverse, ProxyPreserveHost, ProxyReceiveBufferSize, ProxyRemote, ProxyRemoteMatch, ProxyRequests, ProxyTimeout, ProxyVia, RLimitCPU, RLimitMEM, RLimitNPROC, ReadmeName, Redirect, RedirectMatch, RedirectPermanent, RedirectTemp, RemoveCharset, RemoveEncoding, RemoveHandler, RemoveInputFilter, RemoveLanguage, RemoveOutputFilter, RemoveType, RequestHeader, Require, RewriteCond, RewriteRule, SSIEndTag, SSIErrorMsg, SSIStartTag, SSITimeFormat, SSIUndefinedEcho, Satisfy, ScoreBoardFile, Script, ScriptAlias, ScriptAliasMatch, ScriptInterpreterSource, ServerAdmin, ServerAlias, ServerName, ServerPath, ServerRoot, ServerSignature, ServerTokens, SetEnv, SetEnvIf, SetEnvIfNoCase, SetHandler, SetInputFilter, SetOutputFilter, Timeout, TypesConfig, UnsetEnv, UseCanonicalName, XBitHack, allow, deny, order, CGIMapExtension, EnableMMAP, ISAPIAppendLogToErrors, ISAPIAppendLogToQuery, ISAPICacheFile, ISAPIFakeAsync, ISAPILogNotSupported, ISAPIReadAheadBuffer, SSLLog, SSLLogLevel, MaxMemFree, ModMimeUsePathInfo, EnableSendfile, ProxyBadHeader, AllowEncodedSlashes, LimitInternalRecursion, EnableExceptionHook, TraceEnable, ProxyFtpDirCharset, AuthBasicAuthoritative, AuthBasicProvider, AuthDefaultAuthoritative, AuthDigestProvider, AuthLDAPAuthzEnabled, AuthLDAPBindDN, AuthLDAPBindPassword, AuthLDAPCharsetConfig, AuthLDAPCompareDNOnServer, AuthLDAPDereferenceAliases, AuthLDAPGroupAttribute, AuthLDAPGroupAttributeIsDN, AuthLDAPRemoteUserIsDN, AuthLDAPURL, AuthzDBMAuthoritative, AuthzDBMType, AuthzDefaultAuthoritative, AuthzGroupFileAuthoritative, AuthzLDAPAuthoritative, AuthzOwnerAuthoritative, AuthzUserAuthoritative, BalancerMember, DAVGenericLockDB, FilterChain, FilterDeclare, FilterProtocol, FilterProvider, FilterTrace, IdentityCheckTimeout, IndexStyleSheet, ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath, ProxySet, ProxyStatus, ThreadStackSize, AcceptFilter, Protocol, AuthDBDUserPWQuery, AuthDBDUserRealmQuery, UseCanonicalPhysicalPort, CheckCaseOnly, AuthLDAPRemoteUserAttribute, ProxyPassMatch, SSIAccessEnable, Substitute, ProxyPassInterpolateEnv

Htaccess Modules

Here are most of the modules that come with Apache. Each one can have new commands that can be used in .htaccess file scopes.

mod_actions, mod_alias, mod_asis, mod_auth_basic, mod_auth_digest, mod_authn_anon, mod_authn_dbd, mod_authn_dbm, mod_authn_default, mod_authn_file, mod_authz_dbm, mod_authz_default, mod_authz_groupfile, mod_authz_host, mod_authz_owner, mod_authz_user, mod_autoindex, mod_cache, mod_cern_meta, mod_cgi, mod_dav, mod_dav_fs, mod_dbd, mod_deflate, mod_dir, mod_disk_cache, mod_dumpio, mod_env, mod_expires, mod_ext_filter, mod_file_cache, mod_filter, mod_headers, mod_ident, mod_imagemap, mod_include, mod_info, mod_log_config, mod_log_forensic, mod_logio, mod_mem_cache, mod_mime, mod_mime_magic, mod_negotiation, mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_ftp, mod_proxy_http, mod_rewrite, mod_setenvif, mod_speling, mod_ssl, mod_status, mod_substitute, mod_unique_id, mod_userdir, mod_usertrack, mod_version, mod_vhost_alias

Htaccess Software

Apache HTTP Server comes with the following programs.

httpd

Apache hypertext transfer protocol server

apachectl

Apache HTTP server control interface

ab

Apache HTTP server benchmarking tool

apxs

APache eXtenSion tool

dbmmanage

Create and update user authentication files in DBM format for basic authentication

fcgistarter

Start a FastCGI program

htcacheclean

Clean up the disk cache

htdigest

Create and update user authentication files for digest authentication

htdbm

Manipulate DBM password databases.

htpasswd

Create and update user authentication files for basic authentication

httxt2dbm

Create dbm files for use with RewriteMap

logresolve

Resolve hostnames for IP-addresses in Apache logfiles

log_server_status

Periodically log the server’s status

rotatelogs

Rotate Apache logs without having to kill the server

split-logfile

Split a multi-vhost logfile into per-host logfiles

suexec

Switch User For Exec

Here are some example snippets of htaccess files grabbed from various sites.

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteRule ^$ webroot/ [L]
RewriteRule (.*) webroot/$1 [L]
</IfModule>
 
<Limit GET POST PUT>
Order Allow,Deny
Deny from All
</Limit>
 
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteRule ^$ app/webroot/ [L]
RewriteRule (.*) app/webroot/$1 [L]
</IfModule>
 
#
# Apache/PHP/Drupal settings:
#
# Protect files and directories from prying eyes.
 
Options +Indexes
 
AddIcon /icons/compressed.gif .bz2
AddType image/svg+xml .svg
AddType image/svg+xml .svgz
AddType text/html .shtml
AddHandler server-parsed .shtml .html
 
#DirectoryIndex index.php index.html
#Options +FollowSymLinks
#Options -Indexes
 
Options All -Indexes
 
# The concept for this was taken from the Drupal project by D. Keith Casey Jr. - caseydk@use...
# PHP 4, Apache 1
<IfModule mod_php4.c>
php_value magic_quotes_gpc 0
 
Options Indexes
 
XBitHack on
 
<ifmodule mod_rewrite.c>
RewriteEngine On
RewriteRule ^user/(\w+)/(\w+)/(\w+) index.php?user=$1&w=$2&root=$3 [L,QSA]
RewriteRule ^user/(\w+)/(\w+) index.php?user=$1&w=$2 [L,QSA]
RewriteRule ^user/(\w+) index.php?user=$1 [L,QSA]
 
Satisfy All
Options -Indexes
<Files ~ "^\.">
order allow,deny
 
#
# Apache/PHP/Drupal settings:
#
#php_flag display_errors on
#php_flag display_startup_errors on
 
<Files "config.php">
Order Allow,Deny
Deny from All
</Files>
 
<?php
/**
* Smarty plugin
* @package Smarty
* @subpackage plugins
 
DirectoryIndex index.htm index.html index.php index.prc index.htpl
<FilesMatch "\.(htpl|prc)$">
SetHandler htpl-script
</FilesMatch>
 
SetEnv APPLICATION_ENV development
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -s [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
 
<filesmatch "\.php$">
allow from all
</filesmatch>
#Uncomment the below to enable podcasting
 
#
# Apache/PHP/Drupal settings:
#
# Protect files and directories from prying eyes.
AddDefaultCharset utf-8
 
# $Id: .htaccess,v 1.8 2004/11/03 18:21:16 rurban Exp $
<IfModule mod_php4.c>
# For added security (and PhpWiki debugging purposes) you may want to
 
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteRule ^$ webroot/ [L]
RewriteRule (.*) webroot/$1 [L]
</IfModule>
 
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^([0-9]+)$ index.cgi?id=$1 [L]
RewriteRule ^(rss|atom)$ index.cgi?action=qotw&output=$1 [L]
RewriteRule ^([a-z_]+)(/([a-z_]*))?$ index.cgi?action=$1&admin_action=$3 [L,QSA]

More on Page 2.

The Ultimate Htaccess originally appeared on AskApache.com

For debugging, you may want to use my server environment variable debugging script

• Unique mod_setenvif Variables
• Populates HTTP_MY_ Variables with mod_setenvif variable values
• Set REMOTE_HOST to HTTP_HOST
• Allows only if HOST Header is present in request
• Add values from HTTP Headers
• Set the REDIRECT_STATUS for Interpreter Security

Unique mod_setenvif Variables

These can be used for attribute.

Remote_Host

the hostname (if available) of the client making the request – crawl-66-249-70-24.googlebot.com

Remote_Addr

IP address of the client making the request – 66.249.70.24

Server_Addr

IP address of the server on which the request was received – 208.113.183.103

Request_Method

name of the method being used – GET

Request_Protocol

name and version of the protocol with which the request was made – HTTP/1.1

Request_URI

the resource requested on the HTTP request line — generally the portion of the URL following the scheme and host portion without the query string – /robots.txt

Syntax:

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] ...

Populates HTTP_MY_ Variables with mod_setenvif variable values

SetEnvIfNoCase Remote_Host "(.*)" HTTP_MY_REMOTE_HOST=$1
SetEnvIfNoCase Remote_Addr "(.*)" HTTP_MY_REMOTE_ADDR=$1
SetEnvIfNoCase Server_Addr "(.*)" HTTP_MY_SERVER_ADDR=$1
SetEnvIfNoCase Request_Method "(.*)" HTTP_MY_REQUEST_METHOD=$1
SetEnvIfNoCase Request_Protocol "(.*)" HTTP_MY_REQUEST_PROTOCOL=$1
SetEnvIfNoCase Request_URI "(.*)" HTTP_MY_REQUEST_URI=$1

Set REMOTE_HOST to HTTP_HOST

Sets REMOTE_HOST to www.askapache.com if Remote_Addr=208.113.183.103. This can be useful if your server doesn’t automatically do a reverse lookup on a remote address, so this way you can tell if the request was internal/from your server.

SetEnvIf Remote_Addr 208\.113\.183\.103 REMOTE_HOST=www.askapache.com

Allows only if HOST Header is present in request

SetEnvIfNoCase ^HOST$ .+ HTTP_MY_HAS_HOST
Order Deny,Allow
Deny from All
Allow from env=HTTP_MY_HAS_HOST

or

SetEnvIfNoCase Host .+ HTTP_MY_HAS_HOST
Order Deny,Allow
Deny from All
Allow from env=HTTP_MY_HAS_HOST

Add values from HTTP Headers

SetEnvIfNoCase ^If-Modified-Since$ "(.+)" HTTP_IF_MODIFIED_SINCE=$1
SetEnvIfNoCase ^If-None-Match$ "(.+)" HTTP_IF_NONE_MATCH=$1
SetEnvIfNoCase ^Cache-Control$ "(.+)" HTTP_CACHE_CONTROL=$1
SetEnvIfNoCase ^Connection$ "(.+)" HTTP_CONNECTION=$1
SetEnvIfNoCase ^Keep-Alive$ "(.+)" HTTP_KEEP_ALIVE=$1
SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1
SetEnvIfNoCase ^Cookie$ "(.+)" HTTP_MY_COOKIE=$1

Set the REDIRECT_STATUS for Interpreter Security

This is useful in disallowing direct access to interpreters like shell scripts, cgi scripts, and other interpreters. Only works this way if you have a static IP for your server. So the only way to access these files is by instructing the server itself to request the file, using an Action directive or by requesting the file through a .php or other script using curl or wget, or something like fsockopen.

<FilesMatch "\.(cgi|sh|pl)$">
SetEnvIfNoCase Remote_Addr 208\.113\.183\.103 REDIRECT_STATUS
 
Order Deny,Allow
Deny from All
Allow from env=REDIRECT_STATUS
</FilesMatch>

Block Bad Bots

Can be useful if your site is getting hammered by spambots. Some nice examples from around the net are at Fight Blog Spam With Apache… Keep in mind the HTTP_USER_AGENT is directly from the client, so its easy to spoof / change. Instead use mod_security for a much better solution.

SetEnvIfNoCase User-Agent "^Bandit" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole" bad_bot
 
Order Allow,Deny
Allow from All
Deny from env=bad_bot

Allow Search robots

This does the opposite of above, allowing ONLY these web robots access. Other than rogue robots, configuring your robots.txt file correctly will keep most robots where you want them.

SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot
 
Order Deny,Allow
Deny from All
Allow from env=search_robot

SetEnvIf Directive

The SetEnvIf directive defines environment variables based on attributes of the request. The attribute specified in the first argument can be one of three things:

1. An HTTP request header field (see RFC2616 for more information about these); for example: Host, User-Agent, Referer, and Accept-Language. A regular expression may be used to specify a set of request headers.
2. One of the following aspects of the request:
   • Remote_Host – the hostname (if available) of the client making the request
   • Remote_Addr – the IP address of the client making the request
   • Server_Addr – the IP address of the server on which the request was received (only with versions later than 2.0.43)
   • Request_Method – the name of the method being used (GET, POST, et cetera)
   • Request_Protocol – the name and version of the protocol with which the request was made (e.g., “HTTP/0.9″, “HTTP/1.1″, etc.)
   • Request_URI – the resource requested on the HTTP request line — generally the portion of the URL following the scheme and host portion without the query string. See the RewriteCond directive of mod_rewrite for extra information on how to match your query string.
3. The name of an environment variable in the list of those associated with the request. This allows SetEnvIf directives to test against the result of prior matches. Only those environment variables defined by earlier SetEnvIf[NoCase] directives are available for testing in this manner. ‘Earlier’ means that they were defined at a broader scope (such as server-wide) or previously in the current directive’s scope. Environment variables will be considered only if there was no match among request characteristics and a regular expression was not used for the attribute.

The second argument (regex) is a regular expression. If the regex matches against the attribute, then the remainder of the arguments are evaluated.

The rest of the arguments give the names of variables to set, and optionally values to which they should be set. These take the form of

1. varname
2. !varname
3. varname=value

In the first form, the value will be set to “1″. The second will remove the given variable if already defined, and the third will set the variable to the literal value given by value. Since version 2.0.51 Apache will recognize occurrences of $1..$9 within value and replace them by parenthesized subexpressions of regex.

SetEnvIf Example:

SetEnvIf Request_URI "\.gif$" object_is_image=gif
SetEnvIf Request_URI "\.jpg$" object_is_image=jpg
SetEnvIf Request_URI "\.xbm$" object_is_image=xbm
SetEnvIf Referer www\.askapache\.com intra_site_referral
SetEnvIf object_is_image xbm XBIT_PROCESSING=1
SetEnvIf ^SETENVIF*  ^[a-z].*  HAS_SETENVIF

The first three will set the environment variable object_is_image if the request was for an image file, and the fourth sets intra_site_referral if the referring page was somewhere on the www.askapache.com Web site.

The last example will set environment variable HAS_SETENVIF if the request contains any headers that begin with “SETENVIF” whose values begins with any character in the set [a-z].

htaccess Guide Sections

• htaccess tricks for Webmasters
• HTTP Header control with htaccess
• PHP on Apache tips and tricks
• SEO Redirects without mod_rewrite
• mod_rewrite examples, tips, and tricks
• HTTP Caching and Site Speedups
• Authentication on Apache
• htaccess Security Tricks and Tips
• SSL tips and examples
• Variable Fun (mod_env) Section
• .htaccess Security with MOD_SECURITY
• SetEnvIf and SetEnvIfNoCase Examples

« .htaccess Security with MOD_SECURITY | .htaccess Tutorial Index

Htaccess SetEnvIf and SetEnvIfNoCase Examples originally appeared on AskApache.com