WARNING! This is for advanced users of the shell, this is NOT a howto or tutorial. The thing is, I googled how to uninstall cpanel and for once in my life I came up empty.. And certainly the cpanel official sites themselves don't provide any instructions other than to say "Dont uninstall it, reinstall your entire operating system without it.".. they sure don't seem confident that they know exactly what their code is doing. Cpanel is great for most people, perfect for many situations, so don't get me wrong.. this is just for fun.

Why Uninstall?

Why? Because I have always built my servers, php installations, perl installs, ruby, iptables, everything from source. I read the INSTALL/README docs, I read the man pages, and I read the info pages as well. I google for configuration advice, I google for tips, and I don't need a web-based perl script messing my stuff up!

The main problems I had with cpanel, which really is a great bit of software for millions of website developers, is that it was incredibly sneaky! I used it for about 6 months and spent that entire time trying to figure out what the heck it was doing. A couple issues that I really disliked, it takes over your bind install, it takes over your apache install, it takes over your php install. And although it does let you configure some things (very few) for custom configurations and the like, I just don't need any of that. By removing the darn thing I am saving GIGS of space on my server, tons of bandwidth, and most importantly to me I am saving CPU and processing time along with RAM and IO speed.

Anything Else

Please.. I could go on for DAYS! Another reason I have wanted to be rid of cpanel is that I like my servers to be as lean and mean as possible. This means I like as few files and processes as possible. There are many benefits to this, like it's much easier for my integrity checking software and rootkit/antivirus software to run and drastically reduces the false positives. And there is that glaring security issue of constantly having cpanel run it's own software to create the WHM/Cpanel web interface, which is accessible online. I haven't researched cpanel security at all, it's possible that it never has security problems that are published, but for me, why take the chance?

Warning - Caution!

The following is just a quick reference of some of the commands I used to uninstall cpanel. This is for advanced users of the shell. If you aren't that advanced and you run a single one of these commands without fully understanding it, you will probably kill your server, probably lose everything on it permanently, probably not have a website or email for weeks.. So backup all your data FIRST. Also, if you aren't 100% sure you won't run into problems, you should contact your hosts technical support - but be prepared for some MAJOR negativity.. cpanel makes things very easy for hosts, and the last thing tech support wants is to fix a server broken by someone who doesn't know what they are doing.

Last Warning!

Unless you understand what theses commands do and the purpose they serve, do not try any of this. These are not the exact commands I used verbatim, they are also not in order. I only put them up here because I was so amazed that google didn't have any uninstall cpanel intructions. Hopefully it's not a conspiracy that will get my site taken down.. ;)

That said and out of the way, it really only took me about 10 minutes to uninstall cpanel completely. But keep in mind I have been closely monitoring and debugging cpanel for 6 months, so I knew what I was doing. And finally, I do apologize for not having better instructions.. but hey, if you don't get this then you have no business trying to figure out how to uninstall cpanel! It's great software and shouldn't be removed unless you are fully capable of managing email/dns/www/ftp and any/all other servers and services on your machine by hand.

Do This First

I have a few drafts I'm working on at the moment with specifics, but for now you will have to figure it out with google. Basically you want to make sure you don't totally knock your machine offline without being able to reconnect. What I do is compile a static version of openssh and a few other security-type shell tools, and configure this binary sshd to run by using inittab, which is the file run by init (pid 1) and makes sure if it dies it is restarted. Here is my /etc/inittab to run the static sshd binary:

hh:12345:respawn:/failover/os/sbin/aassh -D -q -u0 -f /failover/os/etc/aassh

Another trick is to keep a detached screen logged in to root. That way if you mess up your sudoers or securetty or pam or whatever, you can just reattach and fix it.

Finally, you may want to setup your syslog to start earlier than usual, and set up more than normal verbosity. ( I take it to the max ). Then you should setup a 2nd server or machine somewhere to act as a syslog server. Lastly, configure your web server syslog to copy all messages to the remote syslog you set up. I use a reverse ssh tunnel to encrypt the syslog packets, but when I do something serious like reboot after uninstalling cpanel, I prepare for it by adding additional networking routes on my machine to make sure I will get some logs even if sshd cant start or even if my network addresses aren't brought up correctly.

If that sounds easy to you, please continue. If you are saying: Wha??? Continue in read-only mode.

Also, you can't just uninstall cpanel, I have replaced a lot of cpanel already, like building my own bind, apache, php, syslog and making sure they work and aren't being tampered with by cpanel. Basically cpanel runs everything on your server in most cases, so you should prepare by creating your own static software to replace cpanel, and make sure it works.

Find files Accessing /var/cpanel

More than likely these will need to be killed.

lsof +w -Rg -nP +c15 -x f +D /var/cpanel
lsof +w -Rg -nP +c15 -x f +D /usr/local/cpanel

Killing cpanel

Just an example, your machine may have a lot more than these, I have been slowly taking control of my machine back from cpanel for 6 months, so it was easier for me.

for P in tailwatchd queueprocd cpanellogd exim; do pkill -9 $P; done

Commands and Shortcuts

alias NF='nice find $PWD -mount -depth ! -type d'
alias NFF='nice find $PWD -mount -depth ! -type d | xargs -IF87 file F87'
alias NA='nice find $PWD -mount -depth'
alias NAF='nice find $PWD -mount -depth | xargs -IF87 file F87'

Watch out for crontab

An example of the sneakiness (from my POV, from most it's called builtin robustness) that cpanel does is automagically adding crontab entries that make it behave similarly to a self-propagating virus. If you don't disable the cronjobs and kill the right processes within a short period of time, be prepared for a magic resurrection.

Here's my awesome crontab information function, you will need to check every file, it lists the default crons on my box, and every users crontab, but it can't account for other cron software like at and other crons.

function askapache_crontab()
{
  local GG i;
  for i in `getent passwd|cut -d ":" -f1`;
  do
    GG=$(sudo crontab -u $i -l 2>$N6 | tr -s '\n\000' | sed '/^#/d');
    [[ ${#GG} -gt 3 ]] && sleep 1 && echo -e "$i \n\n${GG}"
  done;
   sleep 4;
   ls -aLls1ch --color=always /etc/cron.{hourly,daily,weekly,monthly,d} | sed '/^total/d; /\ drwxr-xr-x/d';
}

grep -ir /var/spool cpan

#6 3 * * * /scripts/upcp
#0 1 * * * /scripts/cpbackup
#0 2 * * * /scripts/mailman_chown_archives
#35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
#11,26,41,56 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
#30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache
#45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
#*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
#25 1 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify

Delete Crontabs

sudo crontab -u mailman -r

Find INIT scripts with cpanel

This is the main startup script: /usr/local/cpanel/etc/init/startup

I had no idea ruby-on-rails was being controlled by cpanel.. sneaky bugger. You can tell by all of these advanced unix commands just how difficult it would be to uninstall cpanel, its totally like the Alien!

(1:3744)# find . ! -type d -print0|xargs -0 -I'F87' grep -Hi "cpan\|tailwat\|chkser" F87
./fastmail:# Author:       cPanel, Inc. <nick@cpanel.net>
./httpd:        HTTPD=/usr/local/cpanel/bin/chroothttpd
./cpanel:# cpanel8       Start Cpanel Services
./cpanel:# Author:       cPanel, Inc. <nick@cpanel.net>
./cpanel:# description: This is the cpanel webserver and chat.
./cpanel:# processname: cpaneld
./cpanel:# pidfile: /var/run/cpanel.pid
./cpanel:[ -f /usr/local/cpanel/etc/init/startup ] || exit 0
./cpanel:       if [ -f "/var/cpanel/smtpgidonlytweak" ]; then
./cpanel:       echo -n "Starting cPanel services: "
./cpanel:       daemon /usr/local/cpanel/etc/init/startcpsrvd
./cpanel:       echo -n "Starting cPanel brute force detector services: "
./cpanel:       daemon /usr/local/cpanel/etc/init/startcphulkd
./cpanel:    echo -n "Starting cPanel dav services: "
./cpanel:       daemon /usr/local/cpanel/etc/init/startcpdavd
./cpanel:               daemon /usr/local/cpanel/etc/init/startcppop
./cpanel:       echo -n "Starting cPanel Chat services: "
./cpanel:               daemon /usr/local/cpanel/entropychat/entropychat
./cpanel:               daemon /usr/local/cpanel/bin/startmelange
./cpanel:                       /usr/local/cpanel/bin/startinterchange
./cpanel:       echo -n "Starting cPanel ssl services: "
./cpanel:       daemon /usr/local/cpanel/startstunnel
./cpanel:    echo -n "Starting cPanel Queue services: "
./cpanel:       daemon /usr/local/cpanel/etc/init/startqueueprocd
./cpanel:    echo -n "Starting tailwatchd: "
./cpanel:    daemon /usr/local/cpanel/libexec/tailwatchd --start
./cpanel:       echo -n "Starting cPanel Log services: "
./cpanel:       daemon /usr/local/cpanel/cpanellogd
./cpanel:    action "Starting mailman services: " /usr/local/cpanel/etc/init/startmailman
./cpanel:    action "Stopping tailwatchd: " /usr/local/cpanel/libexec/tailwatchd --stop
./cpanel:    action "Stopping cPanel services: " /usr/local/cpanel/etc/init/stopcpsrvd
./cpanel:       action "Stopping cPanel dav services: " /usr/local/cpanel/etc/init/stopcpdavd
./cpanel:       action "Stopping cPanel queue services: " /usr/local/cpanel/etc/init/stopqueueprocd
./cpanel:       action "Stopping cPanel brute force detector services: " /usr/local/cpanel/etc/init/stopcphulkd
./cpanel:               action "Stopping pop3 services: " /usr/local/cpanel/etc/init/stopcppop
./cpanel:       echo -n "Stopping cPanel log services: "
./cpanel:       killproc cpanellogd
./cpanel:       echo -n "Stopping cPanel Chat services: "
./cpanel:       action "Stopping cPanel ssl services: " /usr/local/cpanel/etc/init/stopstunnel
./cpanel:       action "Stopping mailman services: " /usr/local/cpanel/etc/init/stopmailman
./cpanel:       if [ -e "/usr/local/cpanel/3rdparty/mailman/bin/mailmanctl" ]; then
./exim:if [ -e "/etc/chkserv.d" ]; then
./exim:        for file in `ls /etc/chkserv.d`
./exim:            if [ ! -e "/usr/local/cpanel/libexec/tailwatchd" ]; then
./exim:    if [ -x "/usr/local/cpanel/etc/init/startspamd" ]; then
./exim:        /usr/local/cpanel/etc/init/startspamd
./exim:        if [ ! -e "/usr/local/cpanel/libexec/tailwatchd" ]; then
./ror:  /usr/local/cpanel/bin/rormgr --startboot
./ror:  /usr/local/cpanel/bin/rormgr --stopall
./ror:  /usr/local/cpanel/bin/rormgr --stopall
./ror:  /usr/local/cpanel/bin/rormgr --startboot
./ror:  /usr/local/cpanel/bin/rormgr --statusall
./securetmp:# Author:       cPanel, Inc. <copyright@cpanel.net>

Turn off cpanel services

You should remove the below delete command and start by just disabling the inits by turning them off. Then reboot. Then delete. If your machine won't reboot, I told you so, Cpanel told you so, and likely your host told you so.

for S in cpanel ror securetmp fastmail exim; do R=$(command chkconfig --level 123456 $S off ||echo); R=$(command chkconfig --del $S ||echo); done

HTTPD

If you are running chrooted httpd then you'll need to make sure you don't delete your entire webserver on accident. Here's a relevant part from the /etc/init.d/httpd script.

# the path to your httpd binary, including options if necessary
if [ -e "/etc/chroothttpd" ]; then
        HTTPD=/usr/local/cpanel/bin/chroothttpd
else
        HTTPD=/usr/local/apache/bin/httpd
fi

Finding files owned by cpanel

Some super cool bash commands in this post.. let's start with one to find all the files and folders on your machine owned by cpanel. Check your /etc/passwd file for your machines specific usernames and groups. This command saves all the filenames to ~/cpanel-files-backup.txt, which is used by tar next to create a backup of all of them.

{ find / -mount -depth -maxdepth 150 \( -group cpanel -o   -group cpanel-phpmyadmin -o -group cpanel-phppgadmin   -o -group cpanelphpmyadmin   -o -group cpanelphppgadmin   -o -group cpanelhorde   -o -group cpanelroundcube \) -print; find / -mount -depth -maxdepth 150 \( -user cpanel -o   -user cpanel-phpmyadmin -o -user cpanel-phppgadmin   -o -user cpanelphpmyadmin   -o -user cpanelphppgadmin   -o -user cpanelhorde   -o -user cpanelroundcube \) -print; } > ~/cpanel-files-backup.txt

Here's another way to search directories.

 grep --color=always -Hir cpanel /var

Create the Backup

Note that you must have the latest version of tar for this exact command, also you should backup /var/cpanel and /usr/local/cpanel and /etc and heck the whole machine why dontcha!

tar -T ~/cpanel-files-backup.txt -cvz --checkpoint=1000 --checkpoint-action="ttyout=\rHit %s checkpoint #%u" -f /cpanel-files-backup.tgz --totals

Remove Files

Once you do this your upstream without a paddle, you better make sure you know what you're doing with this. This removes all those files.

cat ~/cpanel-files-backup.txt | xargs -I'F87' rm -vfr F87

Additionally you will want to remove /usr/local/cpanel and /var/cpanel - What I always do when running as root is alias my rm command to instead simply move the files to a .trash folder. That way if something goes bork you have a better chance at fixing it.

Find Group-Owned Files

find / -mount -depth -maxdepth 150 \
\(  -group cpanel -o \
  -group cpanel-phpmyadmin \
  -o -group cpanel-phppgadmin \
  -o -group cpanelphpmyadmin \
  -o -group cpanelphppgadmin \
  -o -group mailman \
  -o -group cpanelhorde \
  -o -group cpanelroundcube \
\) -fprintf /root/cpanel-group-files.log '%#8k %#5m %11M %#10u:%-10g %-5U:%-5G %p %f %Y %F\n'

Find User-Owned Files

find / -mount -depth -maxdepth 150 \(
  -user cpanel \
  -o -user cpanel-phpmyadmin \
  -o -user cpanel-phppgadmin \
  -o -user cpanelphpmyadmin \
  -o -user cpanelphppgadmin \
  -o -user mailman \
  -o -user cpanelhorde \
  -o -user cpanelroundcube
\) -fprintf /root/cpanel-users-files.log '%#8k %#5m %11M %#10u:%-10g %-5U:%-5G %p %f %Y %F\n'

       4  0755  drwxr-xr-x     cpanel:cpanel     32002:32004 /var/cpanel/userhomes/cpanel cpanel d reiserfs
       4  0700  drwx------ cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/tmp tmp d reiserfs
       4  0644  -rw-r--r-- cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/.cpanel/caches/featurelists/default.cache default.cache f reiserfs
       4  0700  drwx------ cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/.cpanel/caches/featurelists featurelists d reiserfs
       4  0700  drwx------ cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/.cpanel/caches caches d reiserfs
       4  0700  drwx------ cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/.cpanel .cpanel d reiserfs
       4  0750  drwxr-x--- cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin/mail mail d reiserfs
       4  0711  drwx--x--x cpanel-phpmyadmin:cpanel-phpmyadmin 32005:32007 /var/cpanel/userhomes/cpanel-phpmyadmin cpanel-phpmyadmin d reiserfs
       4  0700  drwx------ cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/sessions sessions d reiserfs
       4  0644  -rw-r--r-- cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/.cpanel/caches/featurelists/default.cache default.cache f reiserfs
       4  0700  drwx------ cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/.cpanel/caches/featurelists featurelists d reiserfs
       4  0700  drwx------ cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/.cpanel/caches caches d reiserfs
       4  0700  drwx------ cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/.cpanel .cpanel d reiserfs
       4  0750  drwxr-x--- cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin/mail mail d reiserfs
       4  0711  drwx--x--x cpanelphppgadmin:cpanelphppgadmin 32009:32011 /var/cpanel/userhomes/cpanelphppgadmin cpanelphppgadmin d reiserfs
       4  0750  drwxr-x--- cpanelroundcube:cpanelroundcube 514  :514   /var/cpanel/userhomes/cpanelroundcube/mail mail d reiserfs
       4  0700  drwx------ cpanelroundcube:cpanelroundcube 514  :514   /var/cpanel/userhomes/cpanelroundcube/sessions sessions d reiserfs
       4  0711  drwx--x--x cpanelroundcube:cpanelroundcube 514  :514   /var/cpanel/userhomes/cpanelroundcube cpanelroundcube d reiserfs
       4  0644  -rw-r--r--     cpanel:cpanel     32002:32004 /var/cpanel/.cpanel/caches/featurelists/default.cache default.cache f reiserfs
       4  0700  drwx------     cpanel:cpanel     32002:32004 /var/cpanel/.cpanel/caches/featurelists featurelists d reiserfs
       4  0700  drwx------     cpanel:cpanel     32002:32004 /var/cpanel/.cpanel/caches caches d reiserfs
       4  0700  drwx------     cpanel:cpanel     32002:32004 /var/cpanel/.cpanel .cpanel d reiserfs
       4  0700  drwx------ cpanelroundcube:cpanelroundcube 514  :514   /var/cpanel/roundcube/tmp tmp d reiserfs
       4  0700  drwx------ cpanelroundcube:cpanelroundcube 514  :514   /var/cpanel/roundcube/log log d reiserfs

Find Permissions

cat ~/cpanel-group-files.log ~/cpanel-users-files.log |tr -s '\000 \t'|cut -d' ' -f3|sort -u

Find files tailwatchd

(1:3732)# $NICE find ${1:-`pwd`} -mount -name '*tailwatch*'
/usr/local/cpanel/libexec/tailwatchd
/usr/local/cpanel/libexec/tailwatch
/usr/local/cpanel/libexec/tailwatch/tailwatchd
/usr/local/cpanel/etc/init/scripts/freebsd/tailwatchd.sh
/usr/local/cpanel/etc/init/scripts/trustix/tailwatchd
/usr/local/cpanel/etc/init/scripts/centos/tailwatchd
/usr/local/cpanel/etc/init/scripts/suse/tailwatchd
/usr/local/cpanel/etc/init/scripts/caos/tailwatchd
/usr/local/cpanel/etc/init/scripts/whitebox/tailwatchd
/usr/local/cpanel/etc/init/scripts/mandrake/tailwatchd
/usr/local/cpanel/etc/init/scripts/debian/tailwatchd
/usr/local/cpanel/etc/init/scripts/redhat/tailwatchd
/usr/local/cpanel/etc/init/scripts/fedora/tailwatchd
/usr/local/cpanel/etc/init/stoptailwatchd
/usr/local/cpanel/etc/init/starttailwatchd
/usr/local/cpanel/bin/tailwatchd
/usr/local/cpanel/logs/tailwatchd_log
/var/log/cpanel/tailwatchd_log
/var/cpanel/log_rotation/cp_tailwatchd_log.cpanellogd
/var/cpanel/tailwatch.positions
/var/run/tailwatchd.pid
/etc/chkserv.d/tailwatchd
/scripts/restartsrv_tailwatchd

Delete cpanel Users/Groups

for U in cpanel-phpmyadmin cpanel-phppgadmin cpanelphpmyadmin cpanelphppgadmin cpanelhorde cpanelroundcube machbuild; do userdel -fr $U; groupdel $U; done

Check for broken symlinks

find / -mount -depth -type l -print0 |xargs -0 -P0 -I'F87' file -s 'F87' | sed -n '/: broken symbolic link to/p'

Especially check /etc

$ find /etc -mount -depth -type l -print0 |xargs -0 -P0 -I'F87' file -s 'F87' | sed -n '/: broken symbolic link to/p'
/etc/ftpd-rsa.pem                   broken symbolic link to `/var/cpanel/ssl/ftp/ftpd-rsa.pem'
/etc/rc.d/rc1.d/K10chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc1.d/K30antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc1.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc3.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc3.d/S80chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc3.d/S80antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc6.d/K10chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc6.d/K30antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc6.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc5.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc5.d/S80chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc5.d/S80antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc2.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc2.d/S80chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc2.d/S80antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc4.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/rc.d/rc4.d/S80chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc4.d/S80antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc0.d/K10chkservd         broken symbolic link to `../init.d/chkservd'
/etc/rc.d/rc0.d/K30antirelayd       broken symbolic link to `../init.d/antirelayd'
/etc/rc.d/rc0.d/K80dcc              broken symbolic link to `../init.d/dcc'
/etc/authlib/authProg               broken symbolic link to `/usr/local/cpanel/bin/courier-auth'

And delete if you are sure

find /etc -mount -depth -type l -print0 |xargs -0 -P0 -I'F87' file -s 'F87' | sed -n '/: broken symbolic link to/p' |cut -d' ' -f1|xargs -I'F87' rm -rvf 'F87'

Reinstall CSF

The only thing I actually used that came with cpanel is the CSF/LFD Firewall package, which is a fantastic piece of software. I had to reinstall this, and to get it working without cpanel add the following line to the csf.conf

GENERIC = "1"

Thats It

Now once you've cleaned up everything, you should try everything conceivable to get an error before rebooting. Like you should start and stop every service in /etc/init.d/, you should use telinit to check various runlevels (which keeps your sshd connection still live). Go all out, should take at least a full hour.

Another thing I like to do is rebuild alot of my source-built software again in case anything got messed up. I upgrade perl from cpanels 5.8.8 to 5.10, which is pretty thorough, and you know, reinstall anything else I think I might need. One of the benefits of compiling your own software is all I have to do is cd to the source directory and type make -B && ( { make test || make check || make checks || make tests; } || echo ) && sudo make install and that's it. The tests/checks are optional of course.

If anyone actually ever reads this and does it, please share your advice here.. everybody knows we need it! Good Luck

HOWTO: Uninstall CPANEL over SSH originally appeared on AskApache.com

Here's an example of validation links from the AskApache footer.

RSS | XHTML 1.1 | CSS 2.1

document.getElementById("validat").innerHTML += ' | <a href="http://feedvalidator.org/check.cgi?url=http://www.askapache.com/feed/">RSS</a>  | <a href="http://validator.w3.org/check/referer?ss=1;outline=1;sp=1;debug">XHTML 1.1</a> | <a href="http://jigsaw.w3.org/css-validator/check/referer?warning=0">CSS 2.1</a>';

Main Javascript Object

Now here's the object that contains all the functions and variables. This is smart to do as a function (all javascript really) because it makes it much safer as it won't conflict with other scripts or having naming collisions. And portable code is always something to strive for.

var AskApacheJS = new Object({
 
  /* Runs on page load.  Initializes ajs as being an alias for AskApacheJS,
  * and runs the append_validation_links function after 5 seconds. */
  _init: function () {
    /*console.log('Running _init');*/
    window.ajs = this;
    setTimeout(ajs.append_validation_links, 5000);
  },
 
  append_validation_links: function () {
    var ac=ajs.gi('validatelinks');
 
    if(ac) {
      var url=encodeURI(window.location).toString().replace(/&/g,"&");
 
      ac.innerHTML =
        "<a href=\"http://jigsaw.w3.org/css-validator/check/referer?warning=0&amp;profile=css21&amp;warning=1\"></a> | "
      + "<a href=\"http://validator.w3.org/feed/check.cgi?url="+url+"\"></a> | "
      + "<a href=\"http://validator.w3.org/check/referer?ss=1;outline=1;sp=1;debug\"></a> | "
      + "<a href=\"http://validator.w3.org/checklink?check=Check&amp;hide_type=all&amp;summary=on&amp;uri="+url+"\"></a> | "
      + "<a href=\"http://www.cynthiasays.com/mynewtester/cynthia.exe?rptmode=-1&amp;url1="+url+"\"></a> | "
      + "<a href=\"http://www.cynthiasays.com/mynewtester/cynthia.exe?rptmode=2&amp;url1="+url+"\"></a>";
    }
  },
 
  /* gi stands for get item, and returns an element by id. Very robust: if it cant find the object it logs to the console. */
  gi: function (B) {
    /*console.log('Running gi '+B);*/
    try {var b = document.getElementById(B);}
    catch(e1) {b = null;try {b = document.all(B);}
    catch(e2) {b = null;try {b = document.layers[B];}
    catch(e3) {b = null;}}}
    return (ajs.isobj(b)) ? b : ajs.er("ERR gi " + B);
  },
 
  /* This function writes the passed msg to the console as type 'error'
  *  and then returns false since it was afterall an error. */
  er: function (msg) { console.error(msg); return false; },
 
  /* Returns true or false based on whether the passed item was an object, used by gi. */
  isobj: function (_1e) { return (typeof _1e != "undefined" && typeof _1e === "object") ? true : false;  }
});

Initialize the Object

;(function () {
  if (!window.ajs)var ajs=AskApacheJS;window.ajs=ajs=AskApacheJS;
  try{ajs._init();}catch(e){AskApacheJS._init();}
})();

FireBug Logging

First things first, this sets up all the logging and debugging needed. This lets you control debugging output displayed in the console used by debuggers like FireBug. Add this to the very top of your javascript.

if(!this["console"]){this.console = {};}
var i=0,tn,cn=["assert","count","debug","dir","dirxml","error","group","groupEnd","info",
"profile","profileEnd","time","timeEnd","trace","warn","log"];
while ((tn = cn[i++])) {
  if (!console[tn]) {
    (function () { var a = tn + "", console[a] = ("log" in console) ?
    function(){var a=tn+"";console[a]=("log" in console) ?
    function(){var b=Array.apply({},arguments);b.unshift(a+":");
    console["log"](b.join(" "))} : function(){}})()
  }
};

Turning On/Off Debugging

It's useful to quickly be able to switch between having the console logging turned on or off. The downside to keeping it on of course is overhead. Here's a very simple method that I use, advanced users use more sophisticated methods than this. Here are the regexes (Adobe CS4 DreamWeaver).

# To turn console logging on
Search for: /\*console.log([^;]*);\*/
Replace with: console.log$1;
 
# To turn console logging off
console.log([^;]*);
/*console.log$1;*/

Console Logging Commented Out

Here's an example of howto log the isobj function from the above object. isobj: function (_1e) { /*@console@*console.log('Running isobj ');**@console@*/ return (typeof _1e != ajs.undefined && typeof _1e === "object") ? true : false; },

Quick and Easy Example

  document.getElementById('validatelinks') && document.getElementById('validatelinks').innerHTML =
        "<a href=\"http://jigsaw.w3.org/css-validator/check/referer?warning=0&amp;profile=css21&amp;warning=1\"></a> | "
      + "<a href=\"http://validator.w3.org/feed/check.cgi?url="+url+"\"></a> | "
      + "<a href=\"http://validator.w3.org/check/referer?ss=1;outline=1;sp=1;debug\"></a> | "
      + "<a href=\"http://validator.w3.org/checklink?check=Check&amp;hide_type=all&amp;summary=on&amp;uri="+url+"\"></a> | "
      + "<a href=\"http://www.cynthiasays.com/mynewtester/cynthia.exe?rptmode=-1&amp;url1="+url+"\"></a> | "
      + "<a href=\"http://www.cynthiasays.com/mynewtester/cynthia.exe?rptmode=2&amp;url1="+url+"\"></a>";

Javascript Snippet to Add HTML, CSS, and FEED Validation Links originally appeared on AskApache.com

Every Apache server comes pre-shipped with an old cgi script named printenv. It's useful and used for printing out the environment variables from your servers point of view. Basically you save this file where you can then call it from your browser, like site.com/printenv.cgi -- then it will be executing as the user running the web server, nobody, apache, etc.. So this tells you all sorts of interesting things about your server, especially if you want/need to know about the users, groups, and file permissions.

Enabling CGI

You need to do 1 of 2 things to use this:

1. Make it executable and in your server directory
2. Use a trick to cause your server to execute it regardless

Htaccess CGI

If you have htaccess then you most likely can get this to work by giving it an execute bit with chmod or from your ftp client, and then use htaccess like this:

Options +ExecCGI Indexes +FollowSymLinks
AddHandler cgi-script .cgi .pl .sh
 
Order Deny,Allow
Deny from All
Allow from 127.0.0.1 ADDYOURIPHERE env=REDIRECT_STATUS
Satisfy All

Search around on this site for php.cgi tricks if you can't seem to get it. Basically you can try stuff like making it an errordocument, using other directives like forcetype, AddType, and even try stuff like mod_security's upload binary setting.

You do not want anyone other than you ever looking at the output of this. It's so full of details about your machine that it would be a huge security problem.

The AskApache Printenv

#!/bin/sh
echo -e "Content-type: text/plain\n\n"
 
# FUNCTIONS
################################################################################################################
function __A ()
{ local __a __i __z;for __a;do __z=\${!${__a}*};for __i in `eval echo "${__z}"`;do echo -e "$__i: ${!__i}";done;done; }
 
function __S ()
{ local L IFS=';';while read -r L;do builtin printf "${#L}@%s\n" "$L";done|sort -n|sed -u 's/^[^@]*//'; }
 
function __P ()
{ local l=`builtin printf %${2:-$__WIDTH}s` && echo -e "${l// /${1:-=}}"; }
 
function __T ()
{ echo -e "\n\n+`__P -`+\n| $*\n+`__P '='`+"; }
 
function __M ()
{ echo -e " >>> $M" $*; }
 
function __H ()
{ command builtin type $1 &>/dev/null && local a="yes" || return 1; }
 
function LE ()
{
 [[ ! -r /proc/${1:-$$}/limits ]] && return;
 sed -e '1z;s/ *$//;:a;$!N;s/\nM.. [a-z]* [a-z]* [a-z]* \{1,\}\([^ ]*\) *\([^ ]*\) *[a-z]* */\1:\2 /;ta;s/u\w\+d/u/g;s/ *$//;s/ / | /g;s/\([^:]\+\):\1/\1:=/g' /proc/${1:-$$}/limits;
}
function LH ()
{
 [[ ! -r /proc/${1:-$$}/limits ]] && return;
 sed -e '1z;s/ *$//;:a;$!N;s/\nM.. \([a-z]\+ [a-z]* [a-z]*\) \{1,\}\([^ ]*\) *\([^ ]*\) *\([a-z]*\) */:\1/;ta;s/ *:/:/g;s/size/sz/g;s/file/f/g;s/ p\w\+y/ pri/g;s/memory/mem/g;s/p\w\+s/procs/g;s/^://;s/:/ | /g' /proc/${1:-$$}/limits
}
 
# CUSTOM SETTINGS
################################################################################################################
__WIDTH=170
LC_COLLATE=C LC_CTYPE=C LC_ALL=C
 
# RUNTIME SETUP
#################################################################################################################
shopt -s dotglob nocaseglob extglob
 
# -C If set, disallow existing regular files to be overwritte
# -f Disable file name generation (globbing
# -e Exit immediately if a command exits with a non-zero status
# -B enable brace expansion
# +H disable History
set -C +f +H -B
 
# make sure we dont create any files
umask 0177
 
# redirect everything to output (no logs or stderr is used)
exec 2>/dev/null
 
# MAIN EXECUTION
#################################################################################################################
{
 __T "EXPANDING PATH"
 {
  __M "ORIG PATH:$PATH"
  PATH=$PATH:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/libexec:/usr/local/apache/bin
  for t in ${PATH//:/ };
  do
   [[ -d "$t" ]] && sed -n -e "/:${t//\//\\/}:/Q1" <<< ":${p:=}:" && p=$p:$t || continue;
  done
  PATH=${p/:/}:.
  __M "NEW PATH:$PATH"
 }
 
 __T "USER INFO"
 {
  __M "UMASK: `(umask 2>/dev/null)` ( `(umask -S 2>/dev/null)` )"
  __H uname && __M "UNAME: `eval echo $(uname -a 2>/dev/null)`"
  __H whoami && __M "WHOAMI: `(whoami 2>/dev/null)`"
  __H id && __M "ID: `(id 2>/dev/null)`"
  __H logname && __M "LOGNAME: `(logname 2>/dev/null)`"
  __H groups && __M "GROUPS: `(groups 2>/dev/null)`"
  echo -e "\n\n"
 }
 
 if __H who;
 then
  __T "LOGGED ON USERS"
  {
   (who -a 2>/dev/null)
  }
 fi;
 
 if [[ -r /etc/passwd ]];
 then
  __T "/etc/passwd"
  {
   (cat /etc/passwd)
  }
 fi;
 
 if __H ulimit;
 then
  __T "USER LIMITS"
  {
   ulimit -a
  }
 fi
 
 if [[ -d /dev ]] && __H ls;
 then
  __T "/dev Directory"
  {
   ( ls -vlaph /dev 2>/dev/null )
  }
 fi;
 
 if [[ -d /proc ]];
 then
  __T "CURRENT PROCESS LIMITS"
  {
   for p in `echo /proc/[0-9]*/limits`
   do
    pid=${p:6:$((${#p}-13))}
    [[ $pid == $PPID || $pid == $$ ]] && continue;
    echo -e "\n/proc/$pid:"
    sed '1s/\x00/ /g;n;s/\x00/\n/g;/.\{2,\}/!d' /proc/$pid/cmdline $p 2>/dev/null
   done
  }
 
  __T "CURRENT PROCESS CMDLINE"
  {
   for p in `echo /proc/[0-9]*/cmdline`;
   do
    pid=${p:6:$((${#p}-13))}
    [[ $pid == $PPID || $pid == $$ ]] && continue;
    __M "[ /proc/$pid ]";
    sed 's/\x00/ /g;G' $p 2>/dev/null
   done
  }
 fi
 
 __T "IP INFORMATION"
 {
  __H ip && __M "IP:" && (ip -o -f inet addr 2>/dev/null) | sed 's/^.*inet \([0-9.]*\).*$/\1/g';
  __H nmap && __M "NMAP:" && (nmap --iflist 2>/dev/null) | sed 1,4d | sed -n '/ethernet/s/^.*) \([0-9.]*\).*$/\1/gp';
  __H ifconfig && __M "IFCONFIG:" && (ifconfig -a 2>/dev/null) | sed -n '/inet a/s/^.*addr:\([0-9.]*\).*$/\1/gp';
  [[ -f "$HOME/.cpanel/datastore/_sbin_ifconfig_-a" ]] && __M "CPANEL CACHE:" && sed -e '/inet/!d; s/.*addr:\([0-9\.]*\).*/\1/g' "$HOME/.cpanel/datastore/_sbin_ifconfig_-a" | sort -u
 }
 
 __T "ROUTE / INTERFACE INFO"
 {
  __H route && __M "ROUTE" && (route -nv 2>/dev/null)
  __H ip && ( ip rule && ip route && ip address ) 2>/dev/null
  __H ifconfig && (ifconfig -a 2>/dev/null)
 }
 
 __T "CGI/1.0 test script report:"
 {
  __A SERVER REQUEST GET SERVER PATH REMOTE AUTH CONTENT HTTP TZ GATEWAY QUERY MO
  echo -e "\n\n"
 }
 
 __T "HIDDEN VARIABLES"
 {
  __A {a..z} {A..Z} _{0..9} _{A..Z} _{a..z} | cat -Tsv 2>/dev/null
  echo -e "\n\n"
 }
 
 __T "DECLARE INFO"
 {
  for i in "r" "i" "a" "x" "t" "-";
  do
   builtin eval declare -$i && echo;
  done | sed 's/^declare //' | cat -Tsv 2>/dev/null
  echo -e "\n\n"
 }
 
 __T "SHELL OPTIONS"
 {
  __A SHELLOPTS BASHOPTS
  echo -e "\$-: $-"
  __P '-' && builtin shopt -s -p
  __P '-' && builtin shopt -u -p
  echo -e "\n\n"
 }
 
 __T "ENV AND EXPORT"
 {
  __H env && command env | cat -Tsv 2>/dev/null && __P '-'
  builtin export | cat -Tsv 2>/dev/null
  echo -e "\n\n"
 }
 
 if __H perl;
 then
  __T "PERL VARIABLES"
  {
   perl -e'foreach $v (sort(keys(%ENV))) {$vv = $ENV{$v};$vv =~ s|\n|\\n|g;$vv =~ s|"|\\"|g;print "${v}=\"${vv}\"\n"}' | cat -Tsv 2>/dev/null
   echo -e "\n\n"
  }
 fi
 
} | fold - -w$(($__WIDTH+3))
exit $?

Enhanced printenv Script for Server Debugging originally appeared on AskApache.com

This is part of the GRUB article on the Arch Linux wiki that I contributed. As of now, just a copy.

The grub menu.lst provides for a convenient way to add a number of entries with extended kernel parameters to configure all sorts of advanced settings to enable you to quickly and conveniently boot into your existing system with varying levels of debugging output. It's very easy and useful to create several levels of debugging just by adding additional entries to your grub configuration. And if you ever have issues or problems down the road due to a power-failure or hardware failure, it can save you hours of trouble, and of course nothing can beat debugging output when it comes to learning about your system.

Useful Menu.lst Entries

If you are interested in debugging, then you deserve some grub entries for powerusers, here are a few that I like (just add to your menu.lst).

title Shutdown the Computer
halt
 
title Reboot the Computer
reboot
 
title Command Line
commandline
 
title Install GRUB to hd0 MBR
root (hd0,0)
setup (hd0)
 
title Matrix
color green/black light-green/green
 
title Scan for /boot/grub/menu.lst
find --set-root --ignore-floppies /boot/grub/menu.lst
configfile /boot/grub/menu.lst
 
title Scan for /boot/menu.lst
find --set-root --ignore-floppies /menu.lst
configfile /boot/menu.lst
 
# http://www.vortex.prodigynet.co.uk/x86test/
title    Run x86test (CPU Info)
kernel /boot/x86test_zImage.bin
#wget http://www.vortex.prodigynet.co.uk/x86test/x86test_zImage.bin
 
# http://www.memtest.org/
title    Run memtest86+ (Memory Testing)
kernel /boot/memtest86+-1.70.bin

Light Debug

A quick way to see more verbose messages on your console is to bootup your normal grub entry after appending verbose to the kernel line. This simple word added to your kernel line turns on more logging thanks to the /etc/rc.sysinit file, which at the top of the file runs:

if /bin/grep -q " verbose" /proc/cmdline; then /bin/dmesg -n 8; fi

Very simple way to get a bit more messages and debug output in your logs.

title  Arch Linux DEBUG Light
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait verbose
initrd /kernel26.img

Medium Debug

This example menu.lst entry turns on real logging that is set by the kernel and not in an init script. Adding the debug kernel parameter to your kernel line is recognized by a lot of linux internals and enables quite a bit of debugging compared to the default.

title Arch Linux DEBUG Medium
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug
initrd /kernel26.img

Heavy Debug

An even more impressive kernel parameter is the ignore_loglevel, which causes the system to ignore any loglevel and keeps the internal loglevel at the maximum debugging level, basically rendering dmesg unable to lower the debug level.

title Arch Linux DEBUG Heavy
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug ignore_loglevel
initrd /kernel26.img

Extreme Debug

If the "Heavy Debug" seemed like a lot of output, thats about 1/2 of the logging that occurs with this example. This does a couple things, it uses the earlyprintk parameter to setup your kernel for "early" "printing" of messages to your "vga" screen. The ,keep just lets it stay on the screen longer. This will let you see logs that normally are hidden due to the boot-up process. This also changes the log buffer length to 10MB, and also instructs that any fatal signals be printed with print_fatal_signals. The last one, sched_debug, you can look up in the very excellent kernel documentation on kernel parameters.

title Arch Linux DEBUG Extreme
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug
initrd /kernel26.img

Insane Debug

The first few debugging examples showed some really nice kernel parameters to turn on really verbose debugging. This kind of debugging is absolutely critical if you want to max out your system or just learn more about what is going on behind the scenes. But there is a final trick that is my favorite, it's the ability to set both environment variables, and more importantly, module parameters at boot.

As an example, here is the boot line that I am using at the moment on an older Dell Desktop, just to illustrate module parameters and environment vars.

title  Arch Linux X-256
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait pause_on_oops=5 panic=60 i915.modeset=1 no_console_suspend ipv6.disable=1 TERM=xterm-256color quiet 5
initrd /kernel26.img

Since it's low on both memory and CPU, I disable ipv6. I also turn on kernel modesetting for the i915 video card, set my terminal to be xterm-256color, and boot straight into X. This lets me use a very optimized arch-linux configuration, amazing how fast thanks to using slim as the login manager, ratpoison as my window manager, and terminal with tmux as my login shell, all from boot, as the pstree shows (plus Synergy!).

init,1
  |-slim,3096
  |   |-X,3098 -nolisten tcp vt07 -auth /var/run/slim.auth
  |   `-ratpoison,3107,askapache
  |       |-terminal,5341 -x sh -c exec /usr/bin/tmux -2 -l -u -q attach -d -t tmux-askapache
  |       |   |-bash,11165
  |       |   |-tmux,5345 -2 -l -u -q attach -d -t tmux-askapache
  |       |   `-{terminal},5346
  |       `-xscreensaver,3113 -no-splash
  |-synergyc,6121,galileo -f --name galileo-fire --restart 10.66.66.2:26666
  |
  `-tmux,5348,askapache -2 -l -u -q attach -d -t tmux-askapache
      |-bash,5351
      |   `-ssh,9969 lug@askapache.com
      `-bash,5868
         `-vim,11149 -p sda1/grub/menu.lst /boot/grub/menu.lst

That kind of optimized system is only possible if you first can figure out your system, by debugging both the kernel as previously illustrated, debugging the init process, and most importantly, by debugging the modules enabled for your system's hardware/firmware/software. Debugging modules is challenging but worth the effort, and then you are able to do some truly insane debugging from grub like the following example, note that the actual grub entry is all on one line, but I split it into 4 lines so you could see it all. This basically turns on every module on this little Dell desktop to be at the absolute max debug level. There is so much logging when I boot this that the system grinds to a halt and is slower than a TI-89 calculator (See Improve Boot Performance).

title  Arch Linux DEBUG INSANE
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait ignore_loglevel debug debug_locks_verbose=1 sched_debug initcall_debug mminit_loglevel=4 udev.log_priority=8
       loglevel=8 earlyprintk=vga,keep log_buf_len=10M print_fatal_signals=1 apm.debug=Y i8042.debug=Y drm.debug=1 scsi_logging_level=1 usbserial.debug=Y
       option.debug=Y pl2303.debug=Y firewire_ohci.debug=1 hid.debug=1 pci_hotplug.debug=Y pci_hotplug.debug_acpi=Y shpchp.shpchp_debug=Y apic=debug
       show_lapic=all hpet=verbose lmb=debug pause_on_oops=5 panic=10 sysrq_always_enabled
initrd /kernel26.img

A couple key items from that grub entry are sysrq_always_enabled which forces on the sysrq magic, which really is a lifesaver when debugging at this level as your machine will freeze/stop-responding sometimes and it's nice to use sysrq to kill all tasks, change the loglevel, unmount all filesystems, or do a hard reboot. Another key parameter is the initcall_debug, which debugs the init process in excruciating detail. Very useful at times. The last parametery I find very useful is the udev.log_priority=8 to turn on udev logging.

Break Into Init

For instance, If you add break=y to your kernel cmdline, init will pause early in the boot process (after loading modules) and launch an interactive sh shell which can be used for troubleshooting purposes. (Normal boot continues after logout.) This is very similar to the shell that shows up if your computer gets turned off before it is able to shutdown properly. But using this parameter lets you enter into this mode differently at will.

title  Arch Linux Init Break
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait break=y
initrd /kernel26.img

Debugging init

This awesome parameter udev.log_priority=8 does the same thing as editing the file /etc/udev/udev.conf except it executes earlier, turning on debugging output for udev. If you want to know your hardware, that is the key parameter right there. Another trick is if you change the /etc/udev/udev.conf to be verbose, then you can make your initrd image include that file to turn on verbose udeb debugging by adding it to your {{Filename|/etc/mkinitcpio.conf} like:

FILES="/etc/modprobe.d/modprobe.conf /etc/udev/udev.conf"

, which on arch is as easy as

# mkinitcpio -p kernel26

Debugging udev is key because the initrd performs a root change at the end of its run to usually launch a program like /sbin/init as part of a chroot, and unless the new file system has a valid /dev directory, udev must be initialized before invoking chroot in order to provide /dev/console.

exec chroot . /sbin/init <dev/console >dev/console 2>&1

So basically, you aren't able to view the logs that are generated before /dev/console is initialized by udev or by a special initrd you compiled yourself. One method the kernel developers use to be able to still get the log messages generated before /dev/console is available is to provide an alternative console that you can enable or disable from grub.

Net Console

If you read through the kernel documentation regarding debugging, you will hear about Netconsole, which can be loaded from the kernel line in GRUB, compiled into your kernel, or loaded at runtime as a module. Having a netconsole entry in your menu.lst is most excellent for debugging slower computers like old laptops or thin-clients. It's easy to use. Just setup a 2nd computer (running arch) to accept syslog requests on a remote port, very fast and quick to do on arch-linux, 1 line to syslog.conf. Then you could use a log-color-parser like ccze to view all syslog logs, or just tail your everything.log. Then on your laptop, boot up and select the netconsole entry from the grub menu, and you will start seeing as much logging as you want on your syslog system. This logging lets you view even earlier log output than is available with the earlyprintk=vga kernel parameter, as netconsole is used by kernel hackers and developers, so it's very powerful.

title  Arch Linux DEBUG Netconsole
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro netconsole=514@10.0.0.2/12:34:56:78:9a:bc debug ignore_loglevel
initrd /kernel26.img

Hijacking cmdline

If you do not have access to GRUB or the kernel boottime cmdline, like on a server or virtual machine, as long as you have root permissions you can still enable this kind of simplistic verbose logging using a neat hack. While you cannot modify the /proc/cmdline even as root, you can place your own cmdline file on top of /proc/cmdline, so that accessing /proc/cmdline actually accesses your file.

For example if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo quiet

So I use a simple sed command to replace quiet with verbose like:

sed 's/ quiet/ verbose/' /proc/cmdline > /root/cmdline

Then I bind mount /root/cmdline so that it becomes /proc/cmdline, using the -n option to mount so that this mount won't be recorded in the systems mtab.

mount -n --bind -o ro /root/cmdline /proc/cmdline

Now if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo verbose

Advanced Linux Debugging using a Bootloader (GRUB) originally appeared on AskApache.com

The php part of this article is based on my Advanced WordPress 404.php article from 2008. Many of the following ideas came out of the research performed to enumerate every single Apache ErrorDocument, including learning how to view the defaults and many cool tricks for htaccess.

The PHP HTTP ErrorDocument Handler

Just save this as /err.php or whatever. The best is to put it in a cgi-bin script-alias directory under your DOCUMENT_ROOT like /cgi-bin/e.php but most people don't know how. That way you can setup some advanced stuff in a /cgi-bin/.htaccess file. If you are interested in locking it down, I recommend reading Securing php.ini and php-cgi with .htaccess.

Advantages and Reasons for Using

Fast, HTTP Protocol Compliance, protection. If you are reading this article, you already know and just want to check out the code!

<?php
ob_start();
@set_time_limit(5);
@ini_set('memory_limit', '64M');
@ini_set('display_errors', 'Off');
error_reporting(0);
 
function print_error_page()
{
 
  $status_reason = array(
  100 => 'Continue',
  101 => 'Switching Protocols',
  102 => 'Processing',
  200 => 'OK',
  201 => 'Created',
  202 => 'Accepted',
  203 => 'Non-Authoritative Information',
  204 => 'No Content',
  205 => 'Reset Content',
  206 => 'Partial Content',
  207 => 'Multi-Status',
  226 => 'IM Used',
  300 => 'Multiple Choices',
  301 => 'Moved Permanently',
  302 => 'Found',
  303 => 'See Other',
  304 => 'Not Modified',
  305 => 'Use Proxy',
  306 => 'Reserved',
  307 => 'Temporary Redirect',
  400 => 'Bad Request',
  401 => 'Unauthorized',
  402 => 'Payment Required',
  403 => 'Forbidden',
  404 => 'Not Found',
  405 => 'Method Not Allowed',
  406 => 'Not Acceptable',
  407 => 'Proxy Authentication Required',
  408 => 'Request Timeout',
  409 => 'Conflict',
  410 => 'Gone',
  411 => 'Length Required',
  412 => 'Precondition Failed',
  413 => 'Request Entity Too Large',
  414 => 'Request-URI Too Long',
  415 => 'Unsupported Media Type',
  416 => 'Requested Range Not Satisfiable',
  417 => 'Expectation Failed',
  422 => 'Unprocessable Entity',
  423 => 'Locked',
  424 => 'Failed Dependency',
  426 => 'Upgrade Required',
  500 => 'Internal Server Error',
  501 => 'Not Implemented',
  502 => 'Bad Gateway',
  503 => 'Service Unavailable',
  504 => 'Gateway Timeout',
  505 => 'HTTP Version Not Supported',
  506 => 'Variant Also Negotiates',
  507 => 'Insufficient Storage',
  510 => 'Not Extended'
  );
 
  $status_msg = array(
  400 => "Your browser sent a request that this server could not understand.",
  401 => "This server could not verify that you are authorized to access the document requested.",
  402 => 'The server encountered an internal error or misconfiguration and was unable to complete your request.',
  403 => "You don't have permission to access %U% on this server.",
  404 => "We couldn't find <acronym title='%U%'>that uri</acronym> on our server, though it's most certainly not your fault.",
  405 => "The requested method is not allowed for the URL %U%.",
  406 => "An appropriate representation of the requested resource %U% could not be found on this server.",
  407 => "An appropriate representation of the requested resource %U% could not be found on this server.",
  408 => "Server timeout waiting for the HTTP request from the client.",
  409 => 'The server encountered an internal error or misconfiguration and was unable to complete your request.',
  410 => "The requested resource %U% is no longer available on this server and there is no forwarding address. Please remove all references to this resource.",
  411 => "A request of the requested method GET requires a valid Content-length.",
  412 => "The precondition on the request for the URL %U% evaluated to false.",
  413 => "The requested resource %U% does not allow request data with GET requests, or the amount of data provided in the request exceeds the capacity limit.",
  414 => "The requested URL's length exceeds the capacity limit for this server.",
  415 => "The supplied request data is not in a format acceptable for processing by this resource.",
  416 => 'Requested Range Not Satisfiable',
  417 => "The expectation given in the Expect request-header field could not be met by this server. The client sent <code>Expect:</code>",
  422 => "The server understands the media type of the request entity, but was unable to process the contained instructions.",
  423 => "The requested resource is currently locked. The lock must be released or proper identification given before the method can be applied.",
  424 => "The method could not be performed on the resource because the requested action depended on another action and that other action failed.",
  425 => 'The server encountered an internal error or misconfiguration and was unable to complete your request.',
  426 => "The requested resource can only be retrieved using SSL. Either upgrade your client, or try requesting the page using https://",
  500 => 'The server encountered an internal error or misconfiguration and was unable to complete your request.',
  501 => "This type of request method to %U% is not supported.",
  502 => "The proxy server received an invalid response from an upstream server.",
  503 => "The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.",
  504 => "The proxy server did not receive a timely response from the upstream server.",
  505 => 'The server encountered an internal error or misconfiguration and was unable to complete your request.',
  506 => "A variant for the requested resource <code>%U%</code> is itself a negotiable resource. This indicates a configuration error.",
  507 => "The method could not be performed.  There is insufficient free space left in your storage allocation.",
  510 => "A mandatory extension policy in the request is not accepted by the server for this resource."
  );
 
  // Get the Status Code
  if (isset($_SERVER['REDIRECT_STATUS']) && ($_SERVER['REDIRECT_STATUS'] != 200))$sc = $_SERVER['REDIRECT_STATUS'];
  elseif (isset($_SERVER['REDIRECT_REDIRECT_STATUS']) && ($_SERVER['REDIRECT_REDIRECT_STATUS'] != 200)) $sc = $_SERVER['REDIRECT_REDIRECT_STATUS'];
  $sc = (!isset($_GET['error']) ? 404 : $_GET['error']);
 
  $sc=abs(intval($sc));
 
  // Redirect to server home if called directly or if status is under 400
  if( ( (isset($_SERVER['REDIRECT_STATUS']) && $_SERVER['REDIRECT_STATUS'] == 200) && (floor($sc / 100) == 3) )
     || (!isset($_GET['error']) && $_SERVER['REDIRECT_STATUS'] == 200)  )
  {
      @header("Location: http://{$_SERVER['SERVER_NAME']}",1,302);
      die();
  }
 
  // Check range of code or issue 500
  if (($sc < 200) || ($sc > 599)) $sc = 500;
 
  // Check for valid protocols or else issue 505
  if (!in_array($_SERVER["SERVER_PROTOCOL"], array('HTTP/1.0','HTTP/1.1','HTTP/0.9'))) $sc = 505;
 
  // Get the status reason
  $reason = (isset($status_reason[$sc]) ? $status_reason[$sc] : '');
 
  // Get the status message
  $msg = (isset($status_msg[$sc]) ? str_replace('%U%', htmlspecialchars(strip_tags(stripslashes($_SERVER['REQUEST_URI']))), $status_msg[$sc]) : 'Error');
 
  // issue optimized headers (optimized for your server)
  @header("{$_SERVER['SERVER_PROTOCOL']} {$sc} {$reason}", 1, $sc);
  if( @php_sapi_name() != 'cgi-fcgi' ) @header("Status: {$sc} {$reason}", 1, $sc);
 
  // A very small footprint for certain types of 4xx class errors and all 5xx class errors
  if (in_array($sc, array(400, 403, 405)) || (floor($sc / 100) == 5))
  {
    @header("Connection: close", 1);
    if ($sc == 405) @header('Allow: GET,HEAD,POST,OPTIONS', 1, 405);
  }
 
  echo "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html>";
  echo "<head>\n<title>{$sc} {$reason}</title>\n<h1>{$reason}</h1>\n<p>{$msg}<br />\n</p>\n";
}
 
function askapache_global_debug()
{
  # http://www.php.net/manual/en/function.array-walk.php#100681
  global $_GET,$_POST,$_ENV,$_SERVER;  $g=array('_ENV','_SERVER','_GET','_POST');
  array_walk_recursive($g, create_function('$n','global $$n;if( !!$$n&&ob_start()&&(print "[ $"."$n ]\n")&&array_walk($$n,
    create_function(\'$v,$k\', \'echo "[$k] => $v\n";\'))) echo "<"."p"."r"."e>".htmlspecialchars(ob_get_clean())."<"."/"."pr"."e>";') );
}
 
print_error_page();
//if($_SERVER['REMOTE_ADDR']=='youripaddress')askapache_global_debug();
echo "</body>\n</html>";
echo ob_get_clean();
exit;
?>

Note: If you are installing this on a non-linux/non-apache machine/server, you will need to read your products documentation for custom error documents. It will work on any machine that can run php.

Htaccess ErrorDocument Tips

The thing is, how do you setup your website to use this php file to be able to handle all those HTTP Status Codes gracefully? You just need to configure your server to use that php file for any Status Codes you want. If you are building an ErrorDocument handling system for a server-wide, multi-site, setup, you will want to instead use the method I use. Instead of using a separate language like PHP, Python, Ruby, Perl, etc, to handle errors, I rely on the very safe and fast SSI method. I detailed the advanced ErrorDocument SSI (.htaccess or httpd.conf)..

If you instead like most of us, you will be setting this up for 1 site, or 1 DOCUMENT_ROOT serving virtual hosts. For that the best method is to modify your .htaccess file.

Using Redirect in .htaccess to trigger Errors

This is one of my all time favorite discoveries from my apache studies. It's documented elsewhere on this site, but if you want to test it out, just request www.askapache.com/show-error-402. Of course the error handling that I have in place is quite nice.

Redirect 400 /show-error-400
Redirect 401 /show-error-401
Redirect 402 /show-error-402
Redirect 403 /show-error-403
Redirect 405 /show-error-405
Redirect 406 /show-error-406
Redirect 407 /show-error-407
Redirect 408 /show-error-408
Redirect 409 /show-error-409
Redirect 410 /show-error-410
Redirect 411 /show-error-411
Redirect 412 /show-error-412
Redirect 413 /show-error-413
Redirect 414 /show-error-414
Redirect 415 /show-error-415
Redirect 416 /show-error-416
Redirect 417 /show-error-417
Redirect 418 /show-error-418
Redirect 419 /show-error-419
Redirect 420 /show-error-420
Redirect 421 /show-error-421
Redirect 422 /show-error-422
Redirect 423 /show-error-423
Redirect 424 /show-error-424
Redirect 425 /show-error-425
Redirect 426 /show-error-426
Redirect 500 /show-error-500
Redirect 501 /show-error-501
Redirect 502 /show-error-502
Redirect 503 /show-error-503
Redirect 504 /show-error-504
Redirect 505 /show-error-505
Redirect 506 /show-error-506
Redirect 507 /show-error-507
Redirect 508 /show-error-508
Redirect 509 /show-error-509
Redirect 510 /show-error-510

Powerful Mod_Rewrite Trick

Here's how to combine the power of mod_rewrites ability to parse requests and environment variables with the above Redirect trick to trigger a specific ErrorDocument based on the query_string parameter error. This trick is only on AskApache.com, very powerful trick if you need to force ErrorDocuments.

RewriteCond %{QUERY_STRING} error=([4|5][0-9][0-9]) [NC]
RewriteCond %{QUERY_STRING} !404
RewriteRule . /show-error-%1 [L]

ErrorDocument Example for .htaccess

So if you save the php file as err.php in your DOCUMENT_ROOT, these are the htaccess commands that will enable its use.

The addition of the ?error=num should be unneccessary on a good linux machine, it's a way for lesser OS's and webhosts to still be able to use errordocuments. Basically Apache handles ErrorDocuments by setting special DEBUGGING variables (Start with REDIRECT_) so it's very easy to determine the STATUS CODE by just viewing $_SERVER['REDIRECT_STATUS']. If a recursive type of redirect is going on, you may see $_SERVER['REDIRECT_REDIRECT_STATUS']. Dumb (consistently) OS's like a Windows server almost always have problems with things like that, because they don't give a hoot about POSIX or standards, why should they when no one can view their code anyway. Here are some more details on the REDIRECT_STATUS and other ways to use these variables.

If you want to learn how to enumerate and view the different variables that are in your Apache environment, I think I have the best tutorial on the planet for how to do this with PHP and mod_rewrite with mod_headers. That article is the basis for anyone who is hired to do mod_rewrites on a new server without root access. I would say that one article will inform you more about mod_rewrite then any other article on this site.

###
# ErrorDocument: In the event of a problem or error, what the server will return to the client. URLs
# can begin with a / for local web-paths (relative to DocumentRoot), or be a full URL which the client
# can resolve. Alternatively, a message can be displayed.  If a malformed request is detected, normal
# request processing will be immediately halted and the internal error message returned.
#
# Prior to version 2.0, messages were indicated by prefixing them with a
# single unmatched double quote character.
#
# The special value default can be used to specify Apache's simple hardcoded message and
# will restore Apache's simple hardcoded message.
#
ErrorDocument 400 /err.php?error=400
ErrorDocument 401 /err.php?error=401
ErrorDocument 402 /err.php?error=402
ErrorDocument 403 /err.php?error=403
ErrorDocument 404 /err.php?error=404
ErrorDocument 405 /err.php?error=405
ErrorDocument 406 /err.php?error=406
ErrorDocument 407 /err.php?error=407
ErrorDocument 408 /err.php?error=408
ErrorDocument 409 /err.php?error=409
ErrorDocument 410 /err.php?error=410
ErrorDocument 411 /err.php?error=411
ErrorDocument 412 /err.php?error=412
ErrorDocument 413 /err.php?error=413
ErrorDocument 414 /err.php?error=414
ErrorDocument 415 /err.php?error=415
ErrorDocument 416 /err.php?error=416
ErrorDocument 417 /err.php?error=417
ErrorDocument 422 /err.php?error=422
ErrorDocument 423 /err.php?error=423
ErrorDocument 424 /err.php?error=424
ErrorDocument 426 /err.php?error=426
ErrorDocument 500 /err.php?error=500
ErrorDocument 501 /err.php?error=501
ErrorDocument 502 /err.php?error=502
ErrorDocument 503 /err.php?error=503
ErrorDocument 504 /err.php?error=504
ErrorDocument 505 /err.php?error=505
ErrorDocument 506 /err.php?error=506
ErrorDocument 507 /err.php?error=507
ErrorDocument 510 /err.php?error=510

PHP to handle HTTP Status Codes for ErrorDocument originally appeared on AskApache.com

I am now about 1 week away from publishing the much-anticipated 4.7 update to the AskApache Password Protection WordPress plugin. It's an upgrade I've been working on for almost 2 years (off and on)! I have been using the new version for quite some time now, and have made a lot of improvements to it, and finally I decided enough users have suffered with the old version. I am very excited for this release, it fixes all known bugs in the older versions, and brings some heavy-duty improvements to all facets of this plugin.. not to mention way better security modules (Lots more COOKIE use) based on code I use with clients.

Plugins for Practice

I wrote 3 plugins to learn the skills I knew I needed for this plugin to do what I envisioned. The first was a plugin that displayed all WP's builtin RewriteRules. The next 2 plugins helped me to learn a great about PHP and WP, so that now my plugins are PHP 4/5 compatible and use PHP classes. Just like all the other code I am into, I am finally achieving 100% E_STRICT conformance. PHP isn't tough at all once you learn the syntax.

AskApache Google 404

This is my favorite plugin. I started developing it in order to learn what I needed to know to continue developing the AskApache PassPro plugin. It's a super-stable plugin and very fast. Check it out and look at the source code if you want to get a preview of the kind of coding used on the AA PassPro 4.7 code.

I use it on every site I develop. Part of the error handling code in that plugin (HTTP Error Handling) is used in the passpro release to provide advanced HTTP 1.1 capabilities and help with the pre-setup testing. Currently it is not working 100% on this site, but the problem is because I use the development version of WP and the latest version breaks wordpress 404 usage (again scribus!).

AskApache Debug Viewer

This is my newest plugin that was created for the same reason as the AskApache 404 plugin, to figure out some things for the PassPro plugin. Mainly, to figure out the best ways to debug php, file permissions, and learn as much as I can to prevent anyone getting locked out of their site with 4.7.

One of the last bits of code I want to add to my 4.7 code, which I have been using for over a year, is super logging capabilities to debug any problems the 4.7 plugin may encounter. The latest release of the Debug Viewer, version 2.0, really has strong logging capabilities, and debugging capabilities. Of course I try to push those capabilities to the max and beyond. Currently the usability isn't the greatest, but if you are interested in debugging it is a goldmine of simple, fast, powerful debugging functions.

Everyone Who had Trouble

To those thousands who inadvertantly locked themselves out of their sites with the PassPro plugin, or even worse shut-down ther sites temporarily, I bet that even though it was annoying, or worse than annoying, you came out of it knowing more about your server, your hosting environment, and it showed you a glimpse of the problems malware face when attempting to crack into a blog protected correctly.

Although part of me knows that there is no better way to learn how to secure your site than dealing with getting locked out yourself, I went all out with making sure this version won't lock you out. Similar to the wordpress plugin bootstrap installation, this version creates a test and verifies it works in a test directory before going live. The new version has the most advanced Apache Version and Apache Module Detection every published online, well not yet.. And if you don't have apache, lighthttpd, or a similar server that supports htaccess, I'm adding a pure-php auth so you can at least still have password protection.. but you can bet I will stronly advise such a user to upgrade their hosting to a Linux+Apache host with skill.

Back to work

I've been extremely swamped at work the past 8 months working on a big project, so I only have days here and there to work on it. I know that I am free either Thursday or Fri this week, so I will have a good shot at getting a BETA release out 4.7.0 one of those days or sometime next week. If you have any ideas for the plugin, nows the time to let me know about it.. If you are a php master (I am not) and would like to help over the next 2 weeks, please email me.

AskApache Password Protection 4.7 Update in 2 Weeks originally appeared on AskApache.com

A large client has a secure website where they assemble and create presentations consisting of a single Table of Contents page with many pdf's attached to it. They use the site to make presentations. This is a big client, a rich client, and they needed a way to guarantee they would be able to access the site. So I got this request:

Can you make an offline version of the Page that is always updated and available for download so we can download the offline version and present using that in case the website is down or more often, in case Internet Access is unavailable?

Update: I used COOKIE based authentication to secure this clients site, so that only logged in users can see anything at all, so how do I enable the curl requests to authenticate as well using the COOKIE of the requesting user in each request made by curl? Just add the users HTTP_COOKIE to the headers array used by curl like so:

array(
...
"Cookie: {$_SERVER['HTTP_COOKIE']}"
)

That now means the scraped version of the page is an exact duplicate that the user is looking at. Very sweet!

I can GET anything

So, here's what I hacked together last night, that is being used today. It's essentially 2 files.

1. A php file that scrapes uses curl to scrape all the urls for the page (favicon, css, images, pdfs, etc..)
2. A simple bash shell script acting as a cgi that creates a zip file of all the urls, and a self-extracting exe file for those without a winzip tool

The PHP File

This is a simple script that is given 2 parameters:

1. The url to scrape
2. The type of download to return

scrapeit.php

<?php
ob_start();
 
/**
* gogeturl2() - grabs a url with curl, and saves it to disk, works for all media types, pdf, js, img, etc.
*
* @return
*/
function gogeturl2($url, $saveto)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
if ($fp = fopen($saveto, 'w')) // {curl_setopt($ch,CURLOPT_STDERR, $fp); curl_setopt($ch,CURLOPT_VERBOSE,1);}
{
  curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
  curl_setopt($ch, CURLOPT_MAXCONNECTS, 4);
  curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
  curl_setopt($ch, CURLOPT_FAILONERROR, 1);
  curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
  curl_setopt($ch, CURLOPT_FILE, $fp);
  curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
  curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3",
    "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
    "Accept-Language: en-us,en;q=0.5",
    "Accept-Encoding: none",
    "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
    "Keep-Alive: 300",
    "Connection: Keep-Alive",
    "Pragma:"));
  $r = curl_exec($ch);
  $ch_info = curl_getinfo($ch);
  if (curl_errno($ch)) error_log(print_r($ch_info, 1) . print_r(curl_errno($ch), 1) . print_r(curl_error($ch), 1));
  else curl_close($ch);
  fclose($fp);
}
}
/**
* gogeturl()  returns the source of the requested url (thanks to accept-encoding: none)
*
* @return
*/
function gogeturl($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_MAXCONNECTS, 4);
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
  "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 GTB6",
  "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
  "Accept-Language: en-us,en;q=0.5",
  "Accept-Encoding: none",
  "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
  "Keep-Alive: 115",
  "Connection: keep-alive",
  "Pragma:"));
$g = curl_exec($ch);
if (curl_errno($ch)) error_log(print_r(array(
'chinfo' => $ch_info,
'curl_errno' => curl_errno($ch),
'curl_error' => curl_error($ch)
), 1));
curl_close($ch);
return $g;
}
 
/**
* _mkdir() makes a directory
*
* @return
*/
function _mkdir($path, $mode = 0755)
{
$old = umask(0);
$res = @mkdir($path, $mode);
umask($old);
return $res;
}
 
/**
* rmkdir()  recursively makes a directory tree
*
* @return
*/
function rmkdir($path, $mode = 0755)
{
return is_dir($path) || (rmkdir(dirname($path), $mode) && _mkdir($path, $mode));
}

The following should be in a couple functions, but I was running on a tight time schedule, and hey this $hitt aint free... wait yes it is, always.

// Ok lets get it on!
// first lets setup some variables
if (!isset($_GET['url']) || empty($_GET['url']))die();
$td = $th = $urls = array();
$FDATE = date("m-d-y-Hms");
$FTMP = '/home/askapache/sites/askapache.com/tmp';
$fetch_url = $_GET['url'];
$fu = parse_url($fetch_url);
$fd = substr($FTMP . $fu['path'], 0, - 1);
$FEXE = "{$fd}-{$FDATE}.exe";
$FZIP = "{$fd}-{$FDATE}.zip";
 
// now this is a shortcut to download the css file and add all the images in it to the img_urls array
$img_urls = array();
$gg = preg_match_all("/url\(([^\)]*?)\)/Ui", gogeturl('https://www.askapache.com/askapache-0128770124.css'), $th);
$imgs = array_unique($th[1]);
foreach($imgs as $img)
{
// only because all the links are relative
$img_urls[] = 'https://www.askapache.com' . $img;
}
 
// now fetch the main page, and assemble an array of all the external resources into the urls array
$gg = preg_match_all("/(background|href)=([\"\'])([^\"\'#]+?)([\"\'])/Ui", gogeturl($fetch_url), $th);
foreach($th[3] as $url)
{
if (strpos($url, '.js') !== false)continue;
if (strpos($url, 'wp-login.php') !== false || $url == 'https://www.askapache.com/') continue;
if (strrpos($url, '/') == strlen($url) - 1)continue;
if (strpos($url, 'https://www.askapache.com/') === false)
{
  if ($url[0] == '/') $urls[] = 'https://www.askapache.com' . $url;
  else continue;
}
else $urls[] = $url;
}
 
// now create a uniq array of urls, then download and save each of them
$urls = array_merge(array_unique($img_urls), array_unique($urls));
foreach($urls as $url)
{
  $pu = parse_url($url);
  rmkdir(substr($fd . $pu['path'], 0, strrpos($fd . $pu['path'], '/')));
  gogeturl2($url, $fd . $pu['path']);
}
 
// deletes dir ie. /this-page/this-page/ when it should be /this-page/index.html
if (is_dir($fd . $fu['path'])) rmdir($fd . $fu['path']);
 
// now save the page as index.html
gogeturl2($fetch_url, $fd . '/index.html');
 
// fixup to be able3 to parse
$g = file_get_contents($fd . '/index.html');
$g = str_replace('<script', '<!--<script', $g);
$g = str_replace('script>', 'script>!-->', $g);
$g = str_replace('href="https://www.askapache.com/', 'href="/', $g);
$g = str_replace('src="https://www.askapache.com/', 'src="/', $g);
$g = str_replace('href="/', 'href="', $g);
$g = str_replace('src="/', 'src="', $g);
$g = str_replace("href='https://www.askapache.com/", "href='/", $g);
$g = str_replace("src='https://www.askapache.com/", "src='/", $g);
$g = str_replace("href='/", "href='", $g);
$g = str_replace("src='/", "src='", $g);
file_put_contents($fd . '/index.html', $g);
 
// fixup for css file
foreach($urls as $url)
{
if (strpos($url, '.css') !== false)
{
  $fuu = parse_url($url);
  $css = file_get_contents($fd . $fuu['path']);
  $css = str_replace('url(/', 'url(../', $css);
  file_put_contents($fd . $fuu['path'], $css);
}
}
 
// my favorite technique, using fsockopen to initiate a shell script server-side.
// passing the args in the HTTP Headers... genius!!
// close the sucker fast with HTTP/1.0 and connection: close
$fp = fsockopen ($_SERVER['SERVER_NAME'], 80, $errno, $errstr, 5);
fwrite($fp, "GET /cgi-bin/sh/zip.sh HTTP/1.0\r\nHost: www.askapache.com\r\nX-Pad: {$fd}\r\nX-Allow: {$FDATE}\r\nConnection: Close\r\n\r\n");
fclose($fp);
 
// loop until the file created by /cgi-bin/sh/zip.sh is found
$c = 0;
do
{
$c++;
sleep(1);
clearstatcache();
if (is_file("{$FEXE}")) continue;
}
while ($c < 20);
 
// either zip or exe
$type = $_GET['type'];
if ($type == 'zip') $file = $FZIP;
else $file = $FEXE;
 
// wow great debugging dude
error_log($file);
 
// if the file is there, do a 302 redirect to initiate download
if (file_exists("{$file}"))
{
  @header("HTTP/1.1 302 Moved", 1, 302);
  @header("Status: 302 Moved", 1, 302);
  @header('Location: https://www.askapache.com/' . basename($file));
  exit;
}
 
exit;
?>

zip.sh

#!/bin/sh
 
# all you need for cgi
  echo -e "Content-type: text/plain\n\n"
 
# blank that run log
  echo "" > /home/askapache/sites/askapache.com/tmp/run.log
 
# redirect 1 and 2 to the run log for the whole script
  exec &>/home/askapache/sites/askapache.com/tmp/run.log
 
# basename
  N=${HTTP_X_PAD//*\/}
 
# date-based
  NN=${HTTP_X_ALLOW}
 
# create recursively the dir tree
  mkdir -pv $HTTP_X_PAD
 
# cd to the tmp
  cd /home/askapache/sites/askapache.com/tmp
 
# the zip version with date
  F=$N-$NN.zip
 
# the exe version with date
  NN=$N-$NN.exe
 
# for debugging, only goes to run log
  echo "F=$F"
  echo "NN=$NN"
  echo "N=$N"
 
# create a relative (r is recursive) archive of the entire dir
  /usr/bin/zip -rvv $F $N
 
# add the self-extracting stub to the archive
  /bin/cat unzipsfx.exe $F > $NN
 
# fix the sfx stub
  /usr/bin/zip -A $NN
 
# move both the exe and zip to the web-docroot to be dl'd directly
  cp -vf $NN /home/askapache/sites/askapache.com/htdocs/
  cp -vf $F /home/askapache/sites/askapache.com/htdocs/
 
# sleep for 60 seconds and then rm all the files, so you better download that file fast
  sleep 60 && rm -rvf $HTTP_X_PAD $F $NN /home/askapache/sites/askapache.com/htdocs/$NN /home/askapache/sites/askapache.com/htdocs/$F
 
#suh suh cya
exit 0;

Creating SFX Archives

The best way is 7z, but I couldn't get p7zip's sfx module to work and didn't have time to compile it. Instead I just used the stub available here: curl -O -L ftp://ftp.info-zip.org/pub/infozip/win32/unz552xn.exe which works great but no customization like password, icons, etc.. oh well.

Implementation

Simple, just create a link to the php file with the url and type parameters. /cgi-bin/php/scrapeit.php?url=http://www.askapache.com/htaccess/htaccess.html&type=exe or if you integrate into wordpress like I did you can add this to your header or admin_bar and use the get_permalink() for the url arg.

Lock It Down

This was used on a private site so what I did was add some code to the scrapeit.php file that just copied the HTTP_COOKIE value sent by the requesting user and sent that as part of the request in the fsockopen request. That means only logged in users can do this, and furthermore, if a user doesn't have access to a page and tries to use this to circumvent, they can't. And also htaccess was used to limit the scripts to only allow the ip's running the server to make connections.

Conclusion

What can't be done with linux, bash, http, php, and a little server-side finesse? My clients are very happy, and I had some fun!

Creating an Offline Version of Page originally appeared on AskApache.com