Mirroring an Entire Site using Rsync over SSH
Sometimes there is an urgent need for creating an exact duplicate or "mirror" of a web site on a separate server. This could be needed for creating Round Robin Setups, Load-Balancing, Failovers, or for just plain vanilla backups. In the past I have used a lot of different methods to copy data from one server to another, including creating an archive of the whole directory and then using scp to send the file over, creating an archive and then encrypting it and then sending that file over using ftp, curl, etc., and my persistence at learning new ways to do things has paid off because now I use rsync to keep an exact replica of the entire directory on an external server, without having to use all the CPU and resources of other mirroring methods.
DNS Round Robin Configuration using Rsync over SSH
The goal is to add the HostGator server to be an exact mirror of the s.askapache.net domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.
Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)
Bash Script to Create index.html of Dir Listing
If you use Apache to auto-generate directory index listings of files/dirs, and you have a large number of files and directories in the root directory and/or slow IO speed, then generating the index could take Apache over a minute!
30x Faster Cache and Site Speed with TMPFS
NOT a typo.. 30x is measurable, well-documented, and easily tested. This is what open-source is about. I haven’t had time to post much the past year, I'm always working! So I wanted to make up for that by publishing an article on a topic that would blow your mind and be something that you could actually start using and really get some benefit out of it. This is one of those articles that the majority of web hosting companies would love to see in paperback, so they could burn it.
PortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG
Ok I just came back up to write the intro.. I'm trying to keep it short to avoid getting bogged down by the coolness of each step. Here is what goes on. When I logon to my XP machine at work, I bring my usb key and plug it in first. On logging a window pops up first and it's a password prompt to mount my encrypted drive leonardo. It also checks a keyfile that is located on my usb key, but all I do now is type in my password. That causes my encrypted folder to be accessible to me like a normal drive, and it autoruns a startup batch file.
The batch file causes Portable versions of Firefox (all my bookmarks, my settings) to load, and launches Portable Mozilla Thunderbird (IMAP makes this work well), which is my favorite program (great GPG features and open-source!). Also Some Adobe CS4 software is loaded from the hard drive, like DreamWeaver. In the background, a service we created executes a PortaPuttY plink command to create forwarded tunnels from various remote servers and accounts, all using key-based encryption. These tunnels are automatically reconnected if they are disconnected, meaning you can use a socks 5 if you want or even better!
Automated Folder Backup Shell-Script
This simple unix shell script automatically creates backups of a specific folder at regular hourly, nightly, weekly, and monthly intervals. Instead of the usual method for copying directory trees using tar with fifo, pipes, rsync, or NFS methods this script uses cpio which is much much faster and has cool options like saving m/a/c times, symlinks, relative paths, and weird file names.
SEO with Robots.txt
Very nice tutorial dealing with the robots.txt file. Shows examples for google and other search engines. Wordpress robots.txt and phpBB robots.txt sample files.Blocking Bad Bots and Scrapers with .htaccess
Want to block a bad robot or web scraper using .htaccess files? Here are 2 methods that illustrate blocking 436 various user-agents.
Optimizing Servers and Processes for Speed with ionice, nice, ulimit
To prepare for several upcoming articles on AskApache that are focused on optimizing Servers and Sites from a server admin level, here is an article to introduce the main tools that we will be using. These tools are used to optimize CPU time for each process using nice and renice, and other tools like ionice are used to optimize the Disk IO, or Disk speed / Disk traffic for each process. Then you can make sure your mysqld and httpd processes are always fast and prioritized.
DreamHost ProcWatch
Share a Mouse and Keyboard between Windows and Linux
Bash alternative to Reflector for Ranking Mirrors
A pure bash alternative to the python reflector, using curl, xargs, and sort for ranking Arch Linux mirrors.
Nice and simple. Short and sweet.
MariaDB with LZ4 Compression Howto for CentOS
I really really wanted to run the latest MariaDB with LZ4 Page Compression.. it is a game changer for many types of large databases I deal with. So I compiled it manually in a way that is repeatable and follows best-practices and it's now powering this site. Now I can use InnoDB Page Compression with lzo, lzma, bzip2, snappy, or my favorite algorithm: LZ4.
Thought this would be a good chance to post a howto, and show that there is a lot you can do by compiling software yourself and breaking the package management one-click install shackles.