Sometimes there is an urgent need for creating an exact duplicate or "mirror" of a web site on a separate server. This could be needed for creating Round Robin Setups, Load-Balancing, Failovers, or for just plain vanilla backups. In the past I have used a lot of different methods to copy data from one server to another, including creating an archive of the whole directory and then using scp to send the file over, creating an archive and then encrypting it and then sending that file over using ftp, curl, etc., and my persistence at learning new ways to do things has paid off because now I use rsync to keep an exact replica of the entire directory on an external server, without having to use all the CPU and resources of other mirroring methods.
Security rsync ssh
A pure bash alternative to the python reflector, using curl, xargs, and sort for ranking Arch Linux mirrors.
Nice and simple. Short and sweet.
Shell Scripting ArchLinux awk bash cURL Python reflector sed shell-script
The goal is to add the HostGator server to be an exact mirror of the s.askapache.net domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.
Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)
Optimization DNS HostGator Rackspace Round-Robin WiredTree
If you use Apache to auto-generate directory index listings of files/dirs, and you have a large number of files and directories in the root directory and/or slow IO speed, then generating the index could take Apache over a minute!
Shell Scripting Apache bash DirectoryIndex Htaccess IndexOptions rsync
NOT a typo.. 30x is measurable, well-documented, and easily tested. This is what open-source is about. I haven’t had time to post much the past year, I'm always working! So I wanted to make up for that by publishing an article on a topic that would blow your mind and be something that you could actually start using and really get some benefit out of it. This is one of those articles that the majority of web hosting companies would love to see in paperback, so they could burn it.
Optimization tmpfs
Want to block a bad robot or web scraper using .htaccess files? Here are 2 methods that illustrate blocking 436 various user-agents.
Htaccess