This is part of the GRUB article on the Arch Linux wiki that I contributed. As of now, just a copy.

The grub menu.lst provides for a convenient way to add a number of entries with extended kernel parameters to configure all sorts of advanced settings to enable you to quickly and conveniently boot into your existing system with varying levels of debugging output. It's very easy and useful to create several levels of debugging just by adding additional entries to your grub configuration. And if you ever have issues or problems down the road due to a power-failure or hardware failure, it can save you hours of trouble, and of course nothing can beat debugging output when it comes to learning about your system.

Useful Menu.lst Entries

If you are interested in debugging, then you deserve some grub entries for powerusers, here are a few that I like (just add to your menu.lst).

title Shutdown the Computer
halt
 
title Reboot the Computer
reboot
 
title Command Line
commandline
 
title Install GRUB to hd0 MBR
root (hd0,0)
setup (hd0)
 
title Matrix
color green/black light-green/green
 
title Scan for /boot/grub/menu.lst
find --set-root --ignore-floppies /boot/grub/menu.lst
configfile /boot/grub/menu.lst
 
title Scan for /boot/menu.lst
find --set-root --ignore-floppies /menu.lst
configfile /boot/menu.lst
 
# http://www.vortex.prodigynet.co.uk/x86test/
title    Run x86test (CPU Info)
kernel /boot/x86test_zImage.bin
#wget http://www.vortex.prodigynet.co.uk/x86test/x86test_zImage.bin
 
# http://www.memtest.org/
title    Run memtest86+ (Memory Testing)
kernel /boot/memtest86+-1.70.bin

Light Debug

A quick way to see more verbose messages on your console is to bootup your normal grub entry after appending verbose to the kernel line. This simple word added to your kernel line turns on more logging thanks to the /etc/rc.sysinit file, which at the top of the file runs:

if /bin/grep -q " verbose" /proc/cmdline; then /bin/dmesg -n 8; fi

Very simple way to get a bit more messages and debug output in your logs.

title  Arch Linux DEBUG Light
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait verbose
initrd /kernel26.img

Medium Debug

This example menu.lst entry turns on real logging that is set by the kernel and not in an init script. Adding the debug kernel parameter to your kernel line is recognized by a lot of linux internals and enables quite a bit of debugging compared to the default.

title Arch Linux DEBUG Medium
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug
initrd /kernel26.img

Heavy Debug

An even more impressive kernel parameter is the ignore_loglevel, which causes the system to ignore any loglevel and keeps the internal loglevel at the maximum debugging level, basically rendering dmesg unable to lower the debug level.

title Arch Linux DEBUG Heavy
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootdelay=5 panic=10 debug ignore_loglevel
initrd /kernel26.img

Extreme Debug

If the "Heavy Debug" seemed like a lot of output, thats about 1/2 of the logging that occurs with this example. This does a couple things, it uses the earlyprintk parameter to setup your kernel for "early" "printing" of messages to your "vga" screen. The ,keep just lets it stay on the screen longer. This will let you see logs that normally are hidden due to the boot-up process. This also changes the log buffer length to 10MB, and also instructs that any fatal signals be printed with print_fatal_signals. The last one, sched_debug, you can look up in the very excellent kernel documentation on kernel parameters.

title Arch Linux DEBUG Extreme
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug
initrd /kernel26.img

Insane Debug

The first few debugging examples showed some really nice kernel parameters to turn on really verbose debugging. This kind of debugging is absolutely critical if you want to max out your system or just learn more about what is going on behind the scenes. But there is a final trick that is my favorite, it's the ability to set both environment variables, and more importantly, module parameters at boot.

As an example, here is the boot line that I am using at the moment on an older Dell Desktop, just to illustrate module parameters and environment vars.

title  Arch Linux X-256
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait pause_on_oops=5 panic=60 i915.modeset=1 no_console_suspend ipv6.disable=1 TERM=xterm-256color quiet 5
initrd /kernel26.img

Since it's low on both memory and CPU, I disable ipv6. I also turn on kernel modesetting for the i915 video card, set my terminal to be xterm-256color, and boot straight into X. This lets me use a very optimized arch-linux configuration, amazing how fast thanks to using slim as the login manager, ratpoison as my window manager, and terminal with tmux as my login shell, all from boot, as the pstree shows (plus Synergy!).

init,1
  |-slim,3096
  |   |-X,3098 -nolisten tcp vt07 -auth /var/run/slim.auth
  |   `-ratpoison,3107,askapache
  |       |-terminal,5341 -x sh -c exec /usr/bin/tmux -2 -l -u -q attach -d -t tmux-askapache
  |       |   |-bash,11165
  |       |   |-tmux,5345 -2 -l -u -q attach -d -t tmux-askapache
  |       |   `-{terminal},5346
  |       `-xscreensaver,3113 -no-splash
  |-synergyc,6121,galileo -f --name galileo-fire --restart 10.66.66.2:26666
  |
  `-tmux,5348,askapache -2 -l -u -q attach -d -t tmux-askapache
      |-bash,5351
      |   `-ssh,9969 lug@askapache.com
      `-bash,5868
         `-vim,11149 -p sda1/grub/menu.lst /boot/grub/menu.lst

That kind of optimized system is only possible if you first can figure out your system, by debugging both the kernel as previously illustrated, debugging the init process, and most importantly, by debugging the modules enabled for your system's hardware/firmware/software. Debugging modules is challenging but worth the effort, and then you are able to do some truly insane debugging from grub like the following example, note that the actual grub entry is all on one line, but I split it into 4 lines so you could see it all. This basically turns on every module on this little Dell desktop to be at the absolute max debug level. There is so much logging when I boot this that the system grinds to a halt and is slower than a TI-89 calculator (See Improve Boot Performance).

title  Arch Linux DEBUG INSANE
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait ignore_loglevel debug debug_locks_verbose=1 sched_debug initcall_debug mminit_loglevel=4 udev.log_priority=8
       loglevel=8 earlyprintk=vga,keep log_buf_len=10M print_fatal_signals=1 apm.debug=Y i8042.debug=Y drm.debug=1 scsi_logging_level=1 usbserial.debug=Y
       option.debug=Y pl2303.debug=Y firewire_ohci.debug=1 hid.debug=1 pci_hotplug.debug=Y pci_hotplug.debug_acpi=Y shpchp.shpchp_debug=Y apic=debug
       show_lapic=all hpet=verbose lmb=debug pause_on_oops=5 panic=10 sysrq_always_enabled
initrd /kernel26.img

A couple key items from that grub entry are sysrq_always_enabled which forces on the sysrq magic, which really is a lifesaver when debugging at this level as your machine will freeze/stop-responding sometimes and it's nice to use sysrq to kill all tasks, change the loglevel, unmount all filesystems, or do a hard reboot. Another key parameter is the initcall_debug, which debugs the init process in excruciating detail. Very useful at times. The last parametery I find very useful is the udev.log_priority=8 to turn on udev logging.

Break Into Init

For instance, If you add break=y to your kernel cmdline, init will pause early in the boot process (after loading modules) and launch an interactive sh shell which can be used for troubleshooting purposes. (Normal boot continues after logout.) This is very similar to the shell that shows up if your computer gets turned off before it is able to shutdown properly. But using this parameter lets you enter into this mode differently at will.

title  Arch Linux Init Break
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro rootwait break=y
initrd /kernel26.img

Debugging init

This awesome parameter udev.log_priority=8 does the same thing as editing the file /etc/udev/udev.conf except it executes earlier, turning on debugging output for udev. If you want to know your hardware, that is the key parameter right there. Another trick is if you change the /etc/udev/udev.conf to be verbose, then you can make your initrd image include that file to turn on verbose udeb debugging by adding it to your {{Filename|/etc/mkinitcpio.conf} like:

FILES="/etc/modprobe.d/modprobe.conf /etc/udev/udev.conf"

, which on arch is as easy as

# mkinitcpio -p kernel26

Debugging udev is key because the initrd performs a root change at the end of its run to usually launch a program like /sbin/init as part of a chroot, and unless the new file system has a valid /dev directory, udev must be initialized before invoking chroot in order to provide /dev/console.

exec chroot . /sbin/init <dev/console >dev/console 2>&1

So basically, you aren't able to view the logs that are generated before /dev/console is initialized by udev or by a special initrd you compiled yourself. One method the kernel developers use to be able to still get the log messages generated before /dev/console is available is to provide an alternative console that you can enable or disable from grub.

Net Console

If you read through the kernel documentation regarding debugging, you will hear about Netconsole, which can be loaded from the kernel line in GRUB, compiled into your kernel, or loaded at runtime as a module. Having a netconsole entry in your menu.lst is most excellent for debugging slower computers like old laptops or thin-clients. It's easy to use. Just setup a 2nd computer (running arch) to accept syslog requests on a remote port, very fast and quick to do on arch-linux, 1 line to syslog.conf. Then you could use a log-color-parser like ccze to view all syslog logs, or just tail your everything.log. Then on your laptop, boot up and select the netconsole entry from the grub menu, and you will start seeing as much logging as you want on your syslog system. This logging lets you view even earlier log output than is available with the earlyprintk=vga kernel parameter, as netconsole is used by kernel hackers and developers, so it's very powerful.

title  Arch Linux DEBUG Netconsole
kernel /vmlinuz26 root=/dev/disk/by-label/ROOT ro netconsole=514@10.0.0.2/12:34:56:78:9a:bc debug ignore_loglevel
initrd /kernel26.img

Hijacking cmdline

If you do not have access to GRUB or the kernel boottime cmdline, like on a server or virtual machine, as long as you have root permissions you can still enable this kind of simplistic verbose logging using a neat hack. While you cannot modify the /proc/cmdline even as root, you can place your own cmdline file on top of /proc/cmdline, so that accessing /proc/cmdline actually accesses your file.

For example if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo quiet

So I use a simple sed command to replace quiet with verbose like:

sed 's/ quiet/ verbose/' /proc/cmdline > /root/cmdline

Then I bind mount /root/cmdline so that it becomes /proc/cmdline, using the -n option to mount so that this mount won't be recorded in the systems mtab.

mount -n --bind -o ro /root/cmdline /proc/cmdline

Now if I cat /proc/cmdline, I have the following:

root=/dev/disk/by-label/ROOT ro console=tty1 logo.nologo verbose

Advanced Linux Debugging using a Bootloader (GRUB) originally appeared on AskApache.com

Back in the mid-90's I was doing a lot of war-dialing with modems (a more recent phenomenon is wardriving, the searching for wireless networks (Wi-Fi) from a moving vehicle. Wardriving was named after wardialing, since both techniques involve brute-force searches to find computer networks) and discovering all sorts of networks and machines, many of them were Unix and Solaris based public systems, and when I managed to gain access to the system and found myself staring at a unix shell I was very excited but also a total idiot. In those days of using the phone networks to research unknown systems it was very difficult for anyone to actually get the phone company to trace a call, so instead of what happens today where it is child's play to trace an IP address, back then it was a very real back-and-forth battle between the system admin and whoever was gaining access to their system.

Essentially, I would gain a shell or some kind of terminal, and just go at it trying to figure out what it could do, trying all kinds of commands. Inevitably this would eventually alert even the laziest admin and they would proceed to attempt to lock me out. It was great sport and extremely addictive. When my favorite system (a massive sun machine in the basement of a big library) finally locked me out and I couldn't get back in I went to my local library and got some reading material -- one of my favorites was the red hat bible. I was able to acquire my own computer and the first thing I did was install red hat linux onto it from the discs included with the book. For the next several years I was essentially offline, all we had at home was a modem and it was becoming difficult to locate any more systems in my area code.. I was into phreaking of course as well, but I never was able to make free long-distance war-dialing a reality.

So I just read the books and learned what I could. I would also goto the library when I could in order to use their machines which were connected to the internet (before aol it was much different than today's internet) and since my time was short I would download as many documents as I could so that I could read them offline. The TLDP documentation that we know today was around back then in various forms, and I read every HOWTO in the index, though not understanding half. The other big resource I found for really intense reading was the kernel documentation, which admitedly I still don't comprehend 1/4th of.. I try and peruse all the new documents when a new kernel is released, since the kernel is where all the real action is, hence the military authoritative name, and that is how I discovered one of the coolest features of Linux that I have found. TMPFS!

What I learned

I learned that with our present technology, which I try to keep somewhat updated on, using a mouse to point and click your way to a solution is always much much much slower than just getting dirty and writing straight code to a terminal. BTW, you know they are close (within 100 years) of enabling connection at the speed of thought. Yes it's a mind-meld between you and a super-computer AI to process it. That's the whole story behind popular Japanese anime like "Ghost in the Shell".

That's also a fundamental reason that certain vast segments of the computing world have a dislike of Microsoft, the way they build (to sell) their operating systems.. no open-code, which is their obvious #1 problem, and why they have a million viruses. Perhaps even a bigger issue some have with "the empire", is how they stuck this gooey, thats how you pronounce GUI - graphical user interface, in a way that tries to create multiple classes of people, those behind the scenes with all the control (they are a middleman between you and your real computer, for millions), and a second group of people on the outside, sheep - I believe they are referred to as sheep.

Cleanup Screenrc Files

You can run this command to get a wildly confusing screenrc file under control. You could have a heavily commented version, and then run this to create the actual... I like to start there but because the screenrc file is read in order, you should take out the 'sort'.

sed -e '/^#/d; s/^[\t\ ]*//g; /^[a-z]/I!d' /etc/screenrc|sort|tr -s ' \t' | tr -d '\015\032'

Read lots of .screenrc files

This is easy with Google's futuristic Code Searching Engine! I get excited about this search engine because it is sooo dope! Here's a search for files named .screenrc, thanks for this free tool Google, much love!

autodetach on
bufferfile $HOME/.screen/buffer
 
nethack on # print wackier status messages
chdir $HOME
password HJa4Dp4UIDlLA
 
setenv LC_CTYPE en_US.UTF-8
sorendition =s Gk
startup_message off
shell bash # use the default shell
shelltitle "$ |bash"
term screen-256color #By default, screen uses an 8-color terminal emulator. Use the following line to enable more colors, more-capable terminal emulator:
 
defmode 620
defmonitor off # turn monitoring on
defnonblock 5 # flaky/slow ssh lines
defscrollback 300
defutf8 on
hardcopydir ~/.hardcopy/screen_%y-%m-%d_%0c.%n
 
defbce on # erase background with current bg color
defencoding UTF-8
defflow auto # will force screen to process ^S/^Q
defflow on # disable Ctrl-S/Ctrl-Q "flow control"
defhstatus "screen: ^En (^Et)"
deflogin off
 
deflog off
logfile ~/.screen/screen_%y-%m-%d_%0c.%n
 
silencewait 15
vbell_msg " Wuff ---- Wuff!! "
 
activity "%C -> %n%f %t activity!"
bell "%C -> %n%f %t bell!~"
 
altscreen on # restore window contents after using (e.g.) vi
attrcolor b ".I" # allow bold colors - necessary for some reason
 
hardstatus on
hardstatus alwayslastline "[%02c] %`%-w%{=b bw}%n %t%{-}%+w %=%{..Lw} %H %{..g} %m/%d %C%a "
caption always "%{kG}%?%-Lw%?%{bw}%n*%f %t%?(%u)%?%{kG}%?%+Lw%?"
 
idle 3600 eval "screen nice -n 19 /opt/s/cmatrix" "idle 6400 detach"
 
msgwait 5 # 1 second messages
multiuser off
nonblock on # If a window goes unresponsive, don't block the whole session waiting for it.
pow_detach_msg "Screen session of $LOGNAME $:cr:$:nl:ended." # emulate .logout message
 
screen -h 500 -t 'bash' sh $HOME/welcome.sh
screen -h 0 -t 'MATRIX' nice -n 19 /opt/s/cmatrix
screen -h 0 -t 'WHO' sudo /opt/s/whowatch
screen -h 100 -t 'gator' sh $HOME/.ssh/start_control.sh
screen -h 100 -t 'gator' ssh gator sh ~/welcome.sh
screen -h 500 -t 'root' sudo bash -l
screen -h 0 -t 'MEM' nice -n 19 sh -c 'sleep 4 && tput civis; CLS=$`tput clear`; trap "tput cnorm; exit 0" 1 2 3; while :; do free -olt && sleep 2 && echo $CLS; done;'
screen -h 0 -t 'TOP' nice -n 19 sh -c 'sleep 4 && tput civis; CLS=$`tput clear`; trap "tput cnorm; exit 0" 1 2 3; while :; do top -b -n 1 -H -d 10 -c |sed "/$USER/!d" && echo $CLS; done;'
screen -h 0 -t 'NET' nice -n 19 sh -c 'while :; do /bin/netstat --numeric-ports -a -e --tcp |sort --key=4 && sleep 5; done;'
select 0
 
#select -h 100 -t 'log' 4 sudo sh -c 'nice tail -n 60 -s 10 -f /var/log/everything.log | ccze -A'
 
zombie "^[" # don't kill window after the process died

More on War Dialing

War dialing or wardialing is a technique of using a modem to automatically scan a list of telephone numbers, usually dialing every number in a local area code to search for computers, Bulletin board systems and fax machines. Hackers use the resulting lists for various purposes, hobbyists for exploration, and crackers - hackers that specialize in computer security - for password guessing.

A single wardialing call would involve calling an unknown number, and waiting for one or two rings, since answering computers usually pick up on the first ring. If the phone rings twice, the modem hangs up and tries the next number. If a modem or fax machine answers, the wardialer program makes a note of the number. If a human or answering machine answers, the wardialer program hangs up. Depending on the time of day, wardialing 10,000 numbers in a given area code might annoy dozens or hundreds of people, some who attempt and fail to answer a phone in two rings, and some who succeed, only to hear the wardialing modem's carrier tone and hang up. The repeated incoming calls are especially annoying to businesses that have many consecutively numbered lines in the exchange, such as used with a Centrex telephone system.

The popularity of wardialing in 1980s and 1990s prompted some states to enact legislation prohibiting the use of a device to dial telephone numbers without the intent of communicating with a person.

The name for this technique originated in the 1983 film WarGames. In the film, the protagonist programmed his computer to dial every telephone number in Sunnyvale, California to find other computer systems. 'WarGames Dialer' programs became common on bulletin board systems of the time, with file names often truncated to wardial.exe and the like due to length restrictions on such systems. Eventually, the etymology of the name fell behind as "war dialing" gained its own currency within computing culture.[1]

A more recent phenomenon is wardriving, the searching for wireless networks (Wi-Fi) from a moving vehicle. Wardriving was named after wardialing, since both techniques involve brute-force searches to find computer networks. The aim of wardriving is to collect information about wireless access points (not to be confused with piggybacking).

Similar to war dialing is a port scan under TCP/IP, which "dials" every TCP port of every IP address to find out what services are available. Unlike wardialing, however, a port scan will generally not disturb a human being when it tries an IP address, regardless of whether there is a computer responding on that address or not. Related to wardriving is warchalking, the practice of drawing chalk symbols in public places to advertise the availability of wireless networks. Despite its widespread coverage [in the news?], warchalking never particularly caught on as a popular activity.

The term is also used today by analogy for various sorts of exhaustive brute force attack against an authentication mechanism, such as a password. While a dictionary attack might involve trying each word in a dictionary as the password, "wardialing the password" would involve trying every possible password. Password protection systems are usually designed to make this impractical, by making the process slow and/or locking out an account for minutes or hours after some low number of wrong password entries.

War dialing is sometimes used as a synonym for demon dialing, a related technique which also involves automating a computer modem in order to repeatedly place telephone calls.

This page contains content by Author of Article from Wikipedia and is licensed under the GNU FDL.

Magic in the Terminal: Screen, Bash, and SSH originally appeared on AskApache.com

Looking for some advanced uses for the shell? Here is some of my best. Works for all shells I encounter, including BackTrack, Debian, Knoppix, Arch Linux, etc. Also works for many hosting environments I use including DreamHost, HostGator, WiredTree, and pretty much any linux VPS.

Want to get started? Type the following in your shell to download the script and run it. You should go over the code first and then you can just save it as your .bash_profile - whenever you want to get updated to the latest version just type aaup, a cool update function that auto dos2unix's and runs via an exec command.

curl -O http://static.askapache.com/askapache-bash-profile.txt && source askapache-bash-profile.txt
Or if you trust me and my hosting provider (and you aren't on production)
curl -o ~/.bash_profile http://static.askapache.com/askapache-bash-profile.txt && exec bash -l

For those of you power users and server admins that use Bash, ksh, csh, vanilla sh, etc.., or if you are just passionate about shell scripting, because it allows you to get advanced tasks done fast and efficiently, not to mention automated automatically. I give you my .bash_profile file. You should edit it to fit your needs, (especially the exported vars like PATH, LDFLAGS, if you don't understand something just comment it out) but it's pretty universal because I work on alot of other people's servers not to mention many different distros and platforms, and when I get hired to do some server work through a shell, I bring this script along for the ride.

askapache-bash-profile.txt

You can download the latest version: http://static.askapache.com/askapache-bash-profile.txt

The functions and variables below are the way bash sees them, using declare -f, and alias, to make it easier for you to read and understand them. The actual file at http://static.askapache.com/askapache-bash-profile.txt will always be the most updated version, as I use it personally. And it has the whole file the way I wrote it, meaning many extra notes and much simpler to follow. Enjoy!

Gist of the Script..

I also rely on this heavily from within shell scripts I write to access all the functions and stuff in this .bash_profile, and to do that I just do like:

#!/bin/bash
 
source ~/.bash_profile &>/dev/nulll
 
pm "PM is a function to output nice messages with color"
yn "Are you enjoying the shell" && pm "Thats great!" || pm "Perhaps you're better suited for DOS"
yn "Show Calendar" && aa_calendar
yn "Show Fortune" && aa_fortune

alias chmod='command chmod -c'
alias cpr='command cp -rpv'
alias df='command df -kTh'
alias df1='command df -iTa'
alias diff='diff -up'
alias dsiz='du -sk * | sort -n --'
alias du='command du -kh'
alias du1='echo *|tr " " "\n" |xargs -iFF command du -hs FF|sort'
alias env='command env | sort'
alias h='history'
alias inice='ionice -c3 -n7 nice'
alias j='jobs -l'
alias la='command ls -Al --color=auto'
alias lc='command ls -lAcr --color=auto'
alias less='vless'
alias lessc='ccze -A |`type -P less` -R'
alias lk='command ls -lASr --color=auto'
alias llh='ll -h'
alias lll='stat -c %a\ %N\ %G\ %U ${PWD}/*|sort'
alias lr='command ls -lAR --color=auto'
alias lt='command ls -lAtr --color=auto'
alias lu='command ls -lAur --color=auto'
alias lx='command ls -lAXB --color=auto'
alias mann='command man -H'
alias n='/usr/bin/nano3'
alias p='command ps -HAcl -F S -A f|uniq -w3'
alias path='echo -e ${PATH//:/\\n}'
alias php='php -d report_memleaks=1 -d report_zend_debug=1 -d log_errors=0 -d ignore_repeated_errors=0 -d ignore_repeated_source=0 -d error_reporting=30719 -d display_startup_errors=1 -d display_errors=1'
alias pp='command ps -HAcl -F S -A f'
alias ps1='command ps -lFA'
alias ps2='command ps -H'
alias resetw='echo $'\''\33[H\33[2J'\'''
alias subash='sudo sh -c '\''export HOME=/root; cd /root; exec bash -l'\'''
alias top='top -c'
alias tree='command tree -Csuflapi'
alias updatedb='( ( updatedb 2>/dev/null ) & )'
alias vim='command vim --noplugin'
alias who='command who -ar -pld'
alias wtf='watch -n 1 w -hs'

Custom bash_profile for Advanced Shell Users originally appeared on AskApache.com

Ok peeps, one of the coolest hacks yet. If you have multiple PC's/Monitors at your workstation like I do, it can be helpful to display various information on one screen while you work on another.

This article shows how I continuously scroll the logs for a server/site I am working on, thus saving me a lot of time by providing real-time debugging on a separate screen. Not only does this scroll the latest log entries as they are created, it displays them in color using syntax highlighting to make your logs incredibly easy to understand and parse.

Example Output

The thing to realize is that this output is continuously scrolling on your monitor, and using some cool linux shell tricks you can make it output at a certain speed and show a certain number of lines. Another cool feature is you can display multiple files at the same time, and the filename will be output for each file above the log output.

What Logs

I've explained in various articles on this site how to create and use various custom log files using .htaccess files and other tricks for non-root users. I use shared hosting all the time too you know... Here are some log files you can generally use if you can use .htaccess files (if you can control .htaccess you can control a binary php-cgi, which could be a shell script thats execs the php binary of your choice with your own environment variables, which is a workaround to getting a custom php.ini, which is how you specify a php error log).

• php error log
• mod_security audit log
• mod_security debug log
• apache error log
• apache access log

Any log file can be used with this method, actually ANY file containing text can be used with this method, even fifo pipes as we will see in a bit ;)

Installation

PCRE

First you need PCRE - Perl Compatible Regular Expressions, which you can download here. Note that I had to install pcre version 6.7 to get it to work with ccze.

CCZE

Now you need CCZE, which colorizes and outputs (emulating tail) the log files.

CCZE is a robust and modular log colorizer with plugins for apm, exim, fetchmail, httpd, postfix, procmail, squid, syslog, ulogd, vsftpd, xferlog, and more.

Post-Install Setup

Ok so once you've installed ccze you can start using it right away, or you can continue reading to see how I've set it up for readable scrolling. If the following setup isn't to your taste, you can always just use wtail, a colorless multi-file scroller that uses separate scrolling windows in your terminal similar to the screen program.

Create Logs Folder

First I created a folder called /logs/ for my site, then for all subsequent commands chdir to it.

$ mkdir /home/site.com/logs
$ cd /home/site.com/logs

Create Symlinks to Logs

In the new logs folder I created soft symlinks to all the various log files, not neccessary but makes everything much easier and organized.

$ ln -s /actual/logs/site.com/http/access.log access.log
$ ln -s /actual/logs/site.com/http/error.log error.log
$ ln -s /home/site.com/php_error.log php_error.log
$ ln -s /home/site.com/modsec_audit.log modsec_audit.log
$ ln -s /home/site.com/modsec_debug.log modsec_debug.log

Make a fifo Pipe

FIFO stands for first-in-first-out and is a somewhat complex feature of linux/bsd/unix shells that let you send data into it and read the data that comes out of it, just like a pipe.

$ mkfifo pipe

Scrolling the Logs

First cd to the log directory you created.

$ cd /home/site.com/logs

Now we will use the tail program to output 120 lines of each of these log files every half of a second, only displaying changes/new log entries. We send that output into the fifo pipe we created using shell redirection and then instruct the command to run in the background using the ampersand to access built-in (bash/sh) job control.

$ tail -s .5 -n 120 -f access.log php_error.log error.log modsec_audit.log >pipe &

Displaying the Colorized Scrolling Output

So every .5 seconds the tail command outputs any new log entries from those files into the fifo pipe, so now we need to use the ccze program to use the fifo pipe as its input log file. Normally you run ccze like the cat command and it just outputs the input colorized, by using tail and a fifo pipe we are able to make this awesome trick work.

To current TTY

$ ccze <pipe

To any TTY

$ ccze < pipe >/dev/pts/2 &

Save All Colored Output

This appends the colorized output to the file in the CWD, super.log

$ ccze -A <pipe | tee -a super.log

Kill the Scrolling

Immediately kills any processes used by tail and ccze.

$ pkill -9 ccze\|tail

Going Further. Hackers only.

Here are some other examples of using ccze with fifo pipes, more as an excercise than anything practical as most lines don't work, I just grabbed them from my shell history file.

tail -f access.log >pipe & ccze <pipe | tee -a super.log >/dev/pts/2
tail -f access.log >pipe & ccze <pipe | tee -a super.log >/dev/pts/2 &
 
exec 3<> pipe; while ccze >/dev/pts/2 <&3; do tail -f access.log >pipe; done; exec 3>&-
exec 3<> pipe; ccze <&3 >/dev/pts/2 & ; tail -f access.log >pipe; exec 3>&-
exec 3<> pipe; ccze <&3 >/dev/pts/2 & ; tail -f access.log >pipe
exec 3<> pipe; ccze <&3 >/dev/pts/2 & tail -f access.log >pipe
ccze <&3 >/dev/pts/2 &
ccze 0<&3 |tee -a super.log > /dev/pts/2 &
ccze <&3 |tee -a super.log > /dev/pts/2 &
ccze <pipe |tee -a /dev/pts/2 & tail -s .5 -n 120 -f php_error.log error.log modsec_audit.log >pipe &
ccze | tee -a /dev/pts/2 <pipe
 
ccze < tee -a /dev/pts/2 pipe >/dev/pts/2
ccze <|tee -a /dev/pts/2 pipe >/dev/pts/2
ccze <tee -a /dev/pts/2 <pipe
ccze <pipe >> tee -a super.log >/dev/pts/2
cat <pipe | ccze
ccze <pipe >>/dev/pts/2 &
 
tail -fq access.log | ccze  &>/dev/pts/1
tail -qf access.log |ccze &>/dev/pts/1 &
tail -f access.log | ccze > $SSH_TTY
 
( ccze > /dev/pts/2 )<pipe
( ccze > /dev/pts/2 )<pipe
( ccze > /dev/pts/2 )<pipe & tail -f access.log | pipe
( ccze > /dev/pts/2 )<pipe & tail -f access.log | ./pipe
 
ccze <pipe > /dev/pts/2 & tail -f access.log >pipe
ccze  <pipe> /dev/pts/2 & tail -f access.log >pipe
ccze  < pipe > /dev/pts/2 & tail -f access.log >pipe
ccze < pipe >/dev/pts/2 & tail -f access.log >pipe
ccze < pipe >/dev/pts/2 & tail -f access.log >pipe &
ccze < pipe >/dev/pts/2 & tail -n 40 -s 2 -f error.log modsec_audit.log php_error.log  >pipe &
ccze < pipe >| tee -a super.log | >/dev/pts/2 & tail -n 80 -s 1 -f error.log modsec_audit.log php_error.log  >pipe &
ccze < pipe >/dev/pts/2 & tail -n 100 -s 1 -f error.log modsec_audit.log php_error.log >pipe &
ccze < pipe >/dev/pts/2 & tail -n 100 -s 1 -f error.log modsec_audit.log php_error.log >pipe &
 
tail -f access.log | ccze > pipe & /dev/pts/2/<pipe
tail -f access.log | ccze > pipe & /dev/pts/2<pipe
 
ccze < $(tail -n 20 -s 1 -f access.log)
ccze < | $(tail -n 20 -s 1 -f access.log)
ccze < |$(tail -n 20 -s 1 -f access.log)

Shell Script

I also hacked together a little shell script that you may wish to hack to your needs. Its actually pretty sweet if you can figure it out.

#!/bin/bash -l
set -o xtrace
renice -p $$ 15
 
pkill -9 tee\|ccze &>/dev/null || echo -n
disown -a &>/dev/null || echo
 
[[ -r "~/pipe" ]] && rm -rf ~/pipe
 
mkfifo ~/pipe
 
cd /home/askapache.com/logs
 
[[ -r "superlog.log" ]] && echo "Found old logfile, moving." && mv superlog.log `command ls|wc -l`-superlog.log
 
ccze <$HOME/pipe > $SSH_TTY & tail -n 150 -s .5 -f  php_error.log error.log modsec_audit.log >$HOME/pipe &
disown %2; && disown %3;
 
disown -a
 
sleep 60 &
disown $! || disown %1
 
for i in `seq 1 4`;do echo -n $'\a'; sleep 1;done
kill -9 $J1
kill -9 $J2
 
logout
exit 0

Elite Log File Scrolling with Color Syntax originally appeared on AskApache.com