Mirroring an Entire Site using Rsync over SSH
Sometimes there is an urgent need for creating an exact duplicate or "mirror" of a web site on a separate server. This could be needed for creating Round Robin Setups, Load-Balancing, Failovers, or for just plain vanilla backups. In the past I have used a lot of different methods to copy data from one server to another, including creating an archive of the whole directory and then using scp to send the file over, creating an archive and then encrypting it and then sending that file over using ftp, curl, etc., and my persistence at learning new ways to do things has paid off because now I use rsync to keep an exact replica of the entire directory on an external server, without having to use all the CPU and resources of other mirroring methods.
Failover SSH for Linux Web Hostmasters
PortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG
Ok I just came back up to write the intro.. I'm trying to keep it short to avoid getting bogged down by the coolness of each step. Here is what goes on. When I logon to my XP machine at work, I bring my usb key and plug it in first. On logging a window pops up first and it's a password prompt to mount my encrypted drive leonardo. It also checks a keyfile that is located on my usb key, but all I do now is type in my password. That causes my encrypted folder to be accessible to me like a normal drive, and it autoruns a startup batch file.
The batch file causes Portable versions of Firefox (all my bookmarks, my settings) to load, and launches Portable Mozilla Thunderbird (IMAP makes this work well), which is my favorite program (great GPG features and open-source!). Also Some Adobe CS4 software is loaded from the hard drive, like DreamWeaver. In the background, a service we created executes a PortaPuttY plink command to create forwarded tunnels from various remote servers and accounts, all using key-based encryption. These tunnels are automatically reconnected if they are disconnected, meaning you can use a socks 5 if you want or even better!
Magic in the Terminal: Screen, Bash, and SSH
Oh ya lets get it on! short but sweet
HOWTO: Uninstall CPANEL over SSH
The following is just a quick but detailed reference of some of the commands I used to successfully uninstall cpanel permanently. This is for advanced users of the shell. If you run a single one of these commands without fully understanding it, you will probably kill your server, lose everything on it permanently, not have a website or email for weeks.. So backup everything. FIRST. You should also contact your hosting provider support - but be prepared for some MAJOR negativity.. cpanel makes things very easy for web-hosting companies, and you are less than a drop in their bucket.
DNS Round Robin Configuration using Rsync over SSH
The goal is to add the HostGator server to be an exact mirror of the s.askapache.net domain, then to add that server as a 2nd A record to my DNS zone. That way half the visitors to the size will be taking up resources and bandwidth on the HostGator server instead of mine.
Round Robin A records in DNS are intended to evenly distribute queries between each host of the same name. Using some tricks straight out of a hackers toolbox we can verify if the distribution is taking place. (It is.)
Crazy POWERFUL Bash Prompt
This amazing bash linux prompt does more than meets the eye. If you want to know how to become really good with technology, linux is the secret sauce behind the AskApache articles. Open Source is elixir of the web. Thanks to everyone who helped me for the past 20 years. I use linux/bsd because homey don't play, so this is geared to be as productive a prompt as I can make it.
Don't have much time.. or just don't care? Not a problem, here are the 3 lines to copy and paste - you can just paste them right in your shell to test it, or add to a startup script.
export AA_P="export PVE=\"\\033[m\\033[38;5;2m\"\$(( \`sed -n \"s/MemFree:[\\t ]\\+\\([0-9]\\+\\) kB/\\1/p\" /proc/meminfo\` / 1024 ))\"\\033[38;5;22m/\"\$((\`sed -n \"s/MemTotal:[\\t ]\\+\\([0-9]\\+\\) kB/\\1/p\" /proc/meminfo\`/ 1024 ))MB\"\\t\\033[m\\033[38;5;55m\$(< /proc/loadavg)\\033[m\";echo -en \"\""
export PROMPT_COMMAND="history -a;((\$SECONDS % 10==0 ))&&eval \"\$AA_P\";echo -en \"\$PVE\";"
export PS1="\\[\\e[m\\n\\e[1;30m\\][\$\$:\$PPID \\j:\\!\\[\\e[1;30m\\]]\\[\\e[0;36m\\] \\T \\d \\[\\e[1;30m\\][\\[\\e[1;34m\\]\\u@\\H\\[\\e[1;30m\\]:\\[\\e[0;37m\\]\${SSH_TTY} \\[\\e[0;32m\\]+\${SHLVL}\\[\\e[1;30m\\]] \\[\\e[1;37m\\]\\w\\[\\e[0;37m\\] \\n(\$SHLVL:\\!)\\\$ " && eval $AA_P
Advanced WordPress wp-config.php Tweaks
The bottom line for this article is that I want to make WordPress as fast, secure, and easy to install, run, and manage because I am using it more and more for client production sites, I will work for days in order to solve an issue so that I never have to spend time on that issue again. Time is money in this industry and that is ultimately (time) what there is to gain by tweaking WordPress.
Note: I spent no time on readability, this is primarily a read the code and figure it out article.. This is for advanced users looking for a reference or discussion and for those of you looking to advance. Feedback would be great if you make it that far..
COMPUTER SECURITY TOOLBOX
List of mainly obscure security software geared more for the master pentester. These are mostly for unix, bsd, and mac and many are difficult to install and setup (require custom servers, inside access points, obscure libraries). Only programs that output data are included, so no actual exploits or anything. Most of these output extremely useful albeit extremely technical information.Upgrading to DreamHost Private Servers
DreamHost PS gives you your own "virtual machine", protecting your CPU and RAM on your physical machine for faster websites. Here's what I like and dislike about DreamHostPS, and some of the issues and solutions for migrating.
Chmod, Umask, Stat, Fileperms, and File Permissions
Unix file permissions are one of the more difficult subjects to grasp.. Well, ok maybe "grasp" isn't the word.. Master is the right word.. Unix file permissions is a hard topic to fully master, mainly I think because there aren't many instances when a computer user encounters them seriously, and bitwise is oldschool. This contains a listing of all possible permission masks and bits from a linux, php, and web hosting view.... cuz you guys AskApache Regs Rock!
Password Protection Plugin Status
Enumerating Permissions can be Annoying
Don’t ask me how because I won’t tell you, but on one of the hosts I was testing on that did not allow direct access I was able to get the Apache server running as dhapache to erroneously write a file into my users blog directory. This is a big security no-no and I now have my .htaccess file written into the blog directory where it should go, but instead of my php script’s user having write access to the file so I can modify it, its owned by dhapache! Because the file is owned by dhapache I shouldn’t even be allowed to know it exists, but there it is. So the next step was to try and take ownership of the .htaccess file so that I could modify it. I tried and tried but was unsuccessful, I couldn’t modify it so that was another dead end. Actually it took me awhile to figure out how to remove the file from my directory. Being that it was owned by dhapache I couldn’t delete or modify it using my php process or even through ftp/ssh! Sysadmins regularly run find commands that search the servers for any files owned by dhapache that should not be there as this is a big red flag that someone has found a way to manipulate dhapache which could potentially lead to modifying dhapache-owned server config files, which sometimes is all it takes to hack your website and server.. Luckily I was able to delete it by basically running the hack again to overwrite the file.
Fix Linux Users Home Permissions with a Cron Job
As a security nut myself, and also a Linux admin, one of my biggest pet peeves is when I've taken the time and care to segment all the users on a server into separate home directories, and then some developer comes along, logs in as root, and changes the ownership of files. Other things can cause this, like Apache, PHP, Mutt, etc.. So I've always used a cron job that executes daily (and on demand) which automatically fixes all the permissions back to what they should be.
Advanced Windows Batch File Scripting
If for some reason you need to use windows .bat batch file scripting to do some task, or you just want to learn the most universal batch programming language on Windows machines, you lucked out and found the AskApache example. My expertise is the shell, in this article the shell is Windows cmd.exe. It has some pretty advanced windows shell usage, including pipes and redirection, but it's the modular linux-like coding approach that earns this script it's "advanced" title.
:SETPROMPT set PROMPT=$_[%USERNAME%@%USERDOMAIN%]$S[$P]$_$M$G && EXIT /B
My favorite tool (and I've tried sooo many) for editing most Windows files and especially .bat files is the free and open-source Notepad++. Set that up and you will have a color-syntax-highlighted editor for Batch Scripting that works very very well.
IP Blocked by stateful Firewall, now what?
Optimizing Servers and Processes for Speed with ionice, nice, ulimit
To prepare for several upcoming articles on AskApache that are focused on optimizing Servers and Sites from a server admin level, here is an article to introduce the main tools that we will be using. These tools are used to optimize CPU time for each process using nice and renice, and other tools like ionice are used to optimize the Disk IO, or Disk speed / Disk traffic for each process. Then you can make sure your mysqld and httpd processes are always fast and prioritized.
Port Redirector
can you please suggest me any tiny and good port redirector for linux and unix like oses ? so that when I connect to the box to a given port it redirects my connection to another box and port that I can chose ? thanx a lotQuestions I Ask Web Hosting Companies, Before Buying
The following is a transcript of a chat I had with a company called tektonic, and at that time I was looking for a cheap linux host to use for some redundancy/failover operations. I generally contact a new hosting company like this every few months.. I like to have options available in case of some kind of failure or network attack, so it's always a good idea to have a few ace linux servers in your back pocket.
If you've read any other articles on AskApache, you can see a certain obsession towards optimization, speed, and security -- so that is the purpose of the following questions.
Libcurl Error Codes
Optimized Vimrc with 256 Colorscheme
This is not your everyday vimrc. This is my personalized vimrc.. works especially great on remote servers, in tmux, screen, etc.
This doesn't require a git checkout, a vim update, a bash update, an OS update, a plugin. No. 1 vimrc, 1 colorscheme, just use curl to download and it's on. I use this on Arch Linux, NetBSD, FreeBSD, OpenBSD, Debian, Ubuntu, Cent OS, Red Hat, and that's mostly it so far.
If you aren't already using vim as your primary editor in a powerful way, you are gonna owe me for this.
Debugging Tools for Windows
I am almost ready to publish the final tutorial on using ssh tunnels from a truecrypt partition on a Windows machine. Of course I choose to go way overboard in my research and the tutorial is full of some pretty awesome windows tricks.. I will come back to this post soon and post all the other advanced tools I use for debugging windows, for now though you MUST know how to debug the kernel and use these basic debugging tools.
Custom Webmaster Shell Environment
Webmasters often use SSH to remotely admin web sites and servers, learn how to make your shell colorful and powerful using my sample .bash_profile and you can be up and running in 30 seconds!
DreamHost ProcWatch
Wget Trick to Download from Restrictive Sites
Before
After trick
I am often logged in to my servers via SSH, and I need to download a file like a WordPress plugin. I've noticed many sites now employ a means of blocking robots like wget from accessing their files. Most of the time they use .htaccess to do this. So a permanent workaround has wget mimick a normal browser.
Elite Log File Scrolling with Color Syntax
Scrolls the latest log entries for multiple log files to the current screen or to any other monitor or TTY in color using syntax highlighting, making debugging easier and saving a lot of time for multi-monitor workstations.
Shell Scripts for Dreamhost
Example shell scripts (bash or sh) that automate and easyate tasks from within the dreamhost ssh env
DreamHost’s Private Servers Finally Rooted
Terminal Escape Code Zen
An image and technicacl achievement so profound, it will touch yoour heart.. So 3D... it'll hurt you eyes! LOL.. lol.. Man I am cracking up here. haha Ha definately the best intro ever, those really do look 3D for terminal though huh.. Sweet. Here is the little function I wrote to output that grey marble.
THE Ultimate Htaccess
.htaccess is a very ancient configuration file for web servers, and is one of the most powerful configuration files most webmasters will ever come across. This htaccess guide shows off the very best of the best htaccess tricks and code snippets from hackers and server administrators.
You've come to the right place if you are looking to acquire mad skills for using .htaccess files!
Firefox Add-ons for Web Developers
Advanced Web Development by AskApache is a Firefox Collection I created since I'm always trying new Addons out and using multiple computers and I wanted a quick and easy way to install my favorite's and keep a running list. Firebug, YSlow, LastPass, and Web Developer are the only ones I always use regularly.
I like the idea of the last.fm but it's not as powerful as the site, which is awesome. Lately listening to Kings of Leon Radio...
Custom bash_profile for Advanced Shell Users
Looking for some advanced uses for the shell? Here is some of my best. The shell is where 70% of my work takes place, and I have at least one terminal open almost 100% of the time, for viewing tailing color-coded logs, and of course for the SSH Tunnels that I use to route various networking through, like my email. So I decided that to standardize and create a bash_profile containing the most time-saving and helpful functions that I could use on all the various hosting environments would really be some sweet sugar, so here is my constant Work-in-progress.
It works for all shells I encounter, including BackTrack, Debian, Knoppix, Arch Linux, etc. Also works for many hosting environments I use including DreamHost, HostGator, WiredTree, and pretty much any linux VPS.
I also rely on this heavily from within shell scripts I write to access all the functions and stuff in this .bash_profile, and to do that I just do like:
#!/bin/bash source ~/.bash_profile &>/dev/nulll pm "PM is a function to output nice messages with color" yn "Are you enjoying the shell" && pm "Thats great!" || pm "Perhaps you're better suited for DOS" yn "Show Calendar" && aa_calendar yn "Show Fortune" && aa_fortune
What I think about DreamHost Web Hosting
NOTE: This is a very old post, I do NOT recommend DreamHost for anyone other than spammers. They are not a long-term solution.
Linux Debugging using a Bootloader with Kernel Parameters
As an example, here is the boot line that I am using at the moment on an older Dell Desktop, just to illustrate module parameters and environment vars.
title Arch Linux X-256 linux /vmlinuz-linux root=UUID rw rootfstype=ext4 elevator=noop selinux=0 plymouth.enable=0 raid=noautodetect ipv6.disable=1 video=DP1:d rcutree.rcu_idle_gp_delay=1 i915.panel_ignore_lid=-1 i915.modeset=1 TERM=rxvt-256color
Bash alternative to Reflector for Ranking Mirrors
A pure bash alternative to the python reflector, using curl, xargs, and sort for ranking Arch Linux mirrors.
Nice and simple. Short and sweet.
King Penguin Linux Notebook
The machine itself is super super ultra-thin, wafer thin, very cool looking. It's crazy light, I was amazed how slick it looked when it arrived.
Huge amount of open-source hardware/chipsets/etc.. The first time I went through the dmesg I was smiling. 2 USB 3.0 ports, and they actually work as promised, very very fast transfer speeds to my USB 3.0 external SSD. Incredible speed, boot time is the fastest I've ever seen. Starts at $720, mine was $1,800
Get Number of Running Proccesses with PHP
Recently I had to setup a script to curl 10k urls, but it could only do 500 requests at any one time. In order to work under that limit, I created a function that returns the number of currently running processes on the machine in an extremely fast and efficient way, thus allowing the curl_multi requests to queu themselves such as GNU xargs.
I got hacked
Well, not this sites server, which has never been hacked, but a server hosted with hostgator.
Interestingly, the hacker sent me an email and left me an mp3, which I've uploaded for your listening leisure.
Do real hackers exist? Yes. Can we coexist? Preferably!