AskApache » Search Results » vulnerability

30x Faster Cache and Site Speed with TMPFS

AskApache — Thu, 18 Mar 2010 15:43:21 +0000

I haven't had time to post much the past year, so I wanted to make up for that by publishing an article on a topic that would blow your mind and be something that you could actually start using and really get some benefit out of it. This is one of those articles that the majority of web hosting companies would love to see in paperback, so they could burn it. Now ask yourself, if a webhost makes money based on how much memory, bandwidth, and data used by a customer, what would they not want their customers to do? That's right, they do not want their customers to learn how to minimize and drastically reduce these moneymakers. They get giddy when you complain about slow-site-speed, or that it takes a long time for your site to load, because they have exactly the right answer- upgrade your memory, bandwidth, and data by purchasing a more expensive plan.

WARNING!! This article has some seriously advanced stuff in it, pretty far beyond my skill level as well (getting there). I personally shutdown some of my own servers with various webhosts because of this.. Note I said personally, not intentionally. Even after spending almost a year (this has been in my drafts folder a long time) using TMPFS on as many machines as I can, I still make mistakes (gotta pay attention!) and lose a tmpfs folder.. Oh and if you go experimenting with this stuff on your web host, you will almost definately, most certainly be on the road to getting your account terminated if you are with one of the cheap hosts. They hate this stuff because it cuts right into the heart of their profit curves and can seriously disrupt a poorly configured machine. DO NOT TRY THIS!! (except and of course on your own development machines). Of course the whole point of this article is how you can take advantage of this incredible filesystem to get crazy speed improvements.. Those are the follow up articles ;)

For those of you who thought modifying your server httpd.conf and htaccess files is very dangerous, you are right. But this is not like that, this is dangerous in the sense that if you try to rush through with your super amazing "copy and paste skills" (script kids) you will easily lose entire folders. That's because TMPFS is stored in RAM/Memory, and upon reboot RAM is cleared. I personally loathe disclaimers, and if you look around you will see there aren't many even with all my sloppy poorly documented articles... So be careful if you feel up to going further.

Introducing tmpfs

If I had to explain tmpfs in one breath, I'd say that tmpfs is like a ramdisk, but different. Like a ramdisk, tmpfs can use your RAM, but it can also use your swap devices for storage. And while a traditional ramdisk is a block device and requires a mkfs command of some kind before you can actually use it, tmpfs is a filesystem, not a block device; you just mount it, and it's there. All in all, this makes tmpfs the niftiest RAM-based filesystem I've had the opportunity to meet.

Beware of WebHosts

What is a modern day web hosting company? What costs do they actually have? A webhost's only unique ability is their connection to the Internet. That is why you can see such tremendous link speed. Other than that they consist of servers that are getting smaller and cheaper for them every month. The servers they use are generally just like any computer, except much larger and built specifically for multi-tasking.

Virtualization allows you to run multiple applications and operating systems independently on a single server. Additionally, administrators can quickly move workloads from one virtual workspace to another — easily prioritizing business needs while maximizing server resources....

Virtualization removes the limitations of the traditional IT approach, enabling a single PowerEdge server to operate multiple applications simultaneously in "virtual machines"

Hosting Company Tricks

Web hosts like to vaguely describe their products as if you are buying your own powerful machine, but in reality you get placed on the same machine as hundreds or thousands of other customers, and the server basically creates an operating system for each customer using virtualization technology. Everyone on the machine literally is sharing the same RAM and resources, many times even sharing IP address's, and the virtualization software lets them limit the amount of memory / cpu / disk / and bandwidth for each of these virtual machines. That is why so often when a web host has an outage they make big public announcements and it appears that hundreds or thousands of their customers have been affected.. One of their server farm machines goes offline and it literally takes down all the customers virtualized machines with it.

Why it gets Evil

Don't get me wrong, I absolutely love this technology, both the hardware virtualization and the software side, but what I truly do not appreciate is how these companies take advantage of their customers every day and know it. Here's what they do, they make justifications about why one plan costs more than another, and these justifications are always about the same thing: CPU's, how fast the data can crunch.. RAM/Memory: How fast and how much your server can handle in terms of traffic... Disk Usage: How much storage you have... And finally bandwidth: How fast can people get data off your sites, and how many people can connect.

Now lets think for a second. The webhost has a BIG computer/server/machine that has MASSIVE amounts of RAM, DISK, PROCESSING power, and NETWORK bandwidth.. but just like anything they all have limits. So if this machine has 10GB of RAM, and the webhost offered plans that have 1GB of RAM, then on that machine they can only have 10 customers right? WRONG. If each customer pays $100/month, then of course they would love to have as many customers on that machine as possible. This builtin incentive is just the reality and isn't anyone fault.

Where it gets Evil

Here's what goes on.. all the host advertises is the 1GB of guaranteed RAM with your machine, but for even if the web server was fairly busy it would never use all of that ram because all the software is careful not to use too much, or has no need for any RAM. Runtime libraries and internal caches use ram, but it's not directly accessed by the customer, only the software. What happens is when those 10 customers aren't using 100% of their ram, which never happens, then the virtualization technology can use that RAM elsewhere. So technically you do have 1GB of RAM available, but if you aren't using it then it is essentially FREE RAM that they can sell to another customer. The only way this wouldn't work of course is if all 11 customers somehow used 100% of RAM simultaneously, at that point the 11th customer would be ramless. But that is impossible because the system is a load-balancing system that provides both an upper and a lower limit to how much RAM is allotted to each virtual machine.

It sounds unrealistic but I see server farms all the time that are stuffed full of virtual machines, like situations where there are 100 1GB customers all sharing 10GB of RAM.. no-one uses the whole 1GB allotted to them as the maximum amount they can use, and they don't know because it appears they have a lot of free RAM, but really that is virtual RAM and could be used by anyone else on the machine.

Where it gets Fun (for me)

This is actually even worse for anyone who is using what they call "shared-hosting" which is the budget hosting that is the most common. With shared-hosting there is actually some skill involved on the hosting companies part, like real linux skills. In this setup they may or more often may not use any virtualization software. It's just a vanilla multi-user server machine where each customer gets a restricted unix account that powers their website using the same system as thousands of others on the box. This is usually dirt cheap because it costs so little to do, but alot of companies charge outrageous amounts for shared-hosting because they make it look really full-featured, which it can be, they just don't mention 1000 other people use the same machine, hard-drive, /tmp directory, network device, IP address, etc.. Alot of the times the cheaper end of the spectrum is where the most gifted system administrators are located, they are so good with linux administration that they could fit 10 customers and 100 websites on an XBOX converted to run linux, and you'd think you got a great deal until you found out! lol. Anyone alive is able to buy more hardware to expand their capacity to take on more customers, but it takes a lot of knowhow and real skill to have that many users on 1 machine. I've seen pretty extreme cases that are analogous to the XBOX example (which is possible by the way).

I personally love shared-hosting environments, because for those of us who know almost as much or more than the system administrators running the machine we are able to use a disproportionate (legally) amount of the CPU and RAM available on the system. So for example my sites would all show up fast and be able to handle more traffic than several other customers combined. Not because anything has been circumvented, but because I am able to access and utilize as much of the guaranteed 1GB of RAM that I am paying for every month, which is usually just a few bucks. The downside is that when you have corporate sites or really high-traffic sites then you are forced to move to a more powerful machine..

This leads to a familiar situation for some of you.. When your site starts becoming popular and you are getting a lot of traffic, this means that your site could be using 10x the amount of RAM and Bandwidth of any other customer in that server farm. And what that really means to the webhost is that you are costing them 10x what anyone else is.. And if they removed you, they would have the space for 10 new customers to take your place, and they would make 10x more money. DreamHost is notorious for terminating accounts because of that.. It happened to me except I was given the option to pay 5x more a month for their "upgrade" to a VPS. Giant shared-hosts advertise like crazy how they offer unlimited bandwidth, but when you start using 100x more bandwidth than anyone on your server you are costing them 100x what you are paying them, every month. That's why you will never see a webhost offering this kind of unlimited bandwidth that doesn't require you to sign a contract giving them permission to terminate your account for any reason. Seriously read the fine print at DreamHost or anywhere else, it's included because that is a core part of their business to terminate anyone using too much bandwidth since that is bandwidth they can't sell to dozens of other customers. That's why I eventually closed my account with them and moved to a legitimate company, it's a great host for spammers though.

Back in the mid-90's I was doing a lot of war-dialing with my modem and discovering all sorts of networks and machines, many of them were Unix and Solaris based public systems, and when I managed to gain access to the system and found myself staring at a unix shell I was very excited but also a total idiot. In those days of using the phone networks to research unknown systems it was very difficult for anyone to actually get the phone company to trace a call, so instead of what happens today where it is child's play to trace an IP address, back then it was a very real back-and-forth battle between the system admin and whoever was gaining access to their system. Essentially, I would gain a shell or some kind of terminal, and just go at it trying to figure out what it could do, trying all kinds of commands. Inevitably this would eventually alert even the laziest admin and they would proceed to attempt to lock me out. It was great sport and extremely addictive. When my favorite system (a massive sun machine in the basement of a big library) finally locked me out and I couldn't get back in I went to my local library and got some reading material -- one of my favorites was the red hat bible. I was able to acquire my own computer and the first thing I did was install red hat linux onto it from the discs included with the book. For the next several years I was essentially offline, all we had at home was a modem and it was becoming difficult to locate any more systems in my area code.. I was into phreaking of course as well, but I never was able to make free long-distance war-dialing a reality. So I just read the books and learned what I could. I would also goto the library when I could in order to use their machines which were connected to the internet (before aol it was much different than today's internet) and since my time was short I would download as many documents as I could so that I could read them offline. The TLDP documentation that we know today was around back then in various forms, and I read every HOWTO in the index, though not understanding half. The other big resource I found for really intense reading was the kernel documentation, which admitedly I still don't comprehend 1/4th of.. I try and peruse all the new documents when a new kernel is released, since the kernel is where all the real action is, hence the military authoritative name, and that is how I discovered one of the coolest features of Linux that I have found. TMPFS!

TMPFS kills the RAMDISK

Ok so we all know what RAM is, it's the memory cards that most people never see that is used by the computer to store and access data that all programs need. RAM is very expensive compared to most PC components, because it's what makes a computer blazing fast or slow. So real quick lets look at a few (there are not many) ways that various linux hackers use RAM in non-conventional ways in the past.

Tmpfs is a file system which keeps all files in virtual memory. Everything is temporary in the sense that no files will be created on your hard drive. If you reboot, everything in tmpfs will be lost.

In contrast to RAM disks, which get allocated a fixed amount of physical RAM, tmpfs grows and shrinks to accommodate the files it contains and is able to swap unneeded pages out to swap space.

Like a ramdisk, tmpfs can use your RAM, but it can also use your swap devices for storage. And while a traditional ramdisk is a block device and requires a mkfs command of some kind before you can actually use it, tmpfs is a filesystem, not a block device; you just mount it, and it's there. All in all, this makes tmpfs the niftiest RAM-based filesystem I've had the opportunity to meet.

What kind of filesystem is used on your server to store all your site files? EXT4, REISERFS, EXT3, NFS, etc.. are the usual filesystems, Windows users are limited to the NTFS filesystem. A filesystem is different than a device, a device is a hard-drive disk. A filesystem is how the device is formatted to allow for file and folder structures. A hard drive is slow compared to RAM, no question about that. So what if instead of your server serving files off a hard-drive it served files stored in RAM? 30x faster thats what happens!

I just figured out how to store my cached static files created by WP-Super Cache in my server's RAM, and the difference is unbelievable. My "AskApache Crazy Cache" plugin basically forces WP-Super Cache, Hyper Cache, etc.. to recreate a static cached file for every page on a blog. For the AskApache.com site this takes around 3 minutes to complete. Once I switched to using this new method of storing the files on RAM I am able to re-cache the entire site in about 15 seconds!!!!

tmpfs is a dynamically expandable/shrinkable ramdisk, and will # use almost no memory if not populated with files

Tmpfs is a file system which keeps all files in virtual memory.

Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost.

tmpfs puts everything into the kernel internal caches and grows and shrinks to accommodate the files it contains and is able to swap unneeded pages out to swap space. It has maximum size limits which can be adjusted on the fly via 'mount -o remount ...'

If you compare it to ramfs (which was the template to create tmpfs) you gain swapping and limit checking. Another similar thing is the RAM disk (/dev/ram*), which simulates a fixed size hard disk in physical RAM, where you have to create an ordinary filesystem on top. Ramdisks cannot swap and you do not have the possibility to resize them.

Since tmpfs lives completely in the page cache and on swap, all tmpfs pages currently in memory will show up as cached. It will not show up as shared or something like that. Further on you can check the actual RAM+swap use of a tmpfs instance with df(1) and du(1).

Both tmpfs and ramfs mount will give you the power of fast reading and writing files from and to the primary memory. When you test this on a small file, you may not see a huge difference. You’ll notice the difference only when you write large amount of data to a file with some other processing overhead such as network.

TMPFS uses RAM+SWAP

TMPFS is another filesystem with uniquely cool capabilities. It stores any files contained within it on RAM and in SWAP which means your server can access any files stored on TMPFS without even having to access the disk, which according to technical stats is around 30 times faster than accessing a file off disk.

Some other cool aspects of TMPFS are that it intelligently and automatically sizes itself to be just alittle bigger then it needs to be. So when you remove files to a folder stored on a TMPFS filesystem, the TMPFS filesystem shrinks by allocating less RAM and/or SWAP. Conversely when adding files to TMPFS it grows larger. You can set the max-size and max-number-of-files as a mount option to make sure your TMPFS never uses all of the available RAM and SWAP, which would halt your server.

Swap

Find the swap size.

# free -m -t
             total       used       free     shared    buffers     cached
Mem:           458         93        364          0          0          0
-/+ buffers/cache:         93        364
Swap:          900          0        900
Total:        1358         93       1264

Adding 3004144k swap on /dev/sdb2.  Priority:-1 extents:1 across:3004144k
Adding 2096472k swap on /dev/sda3.  Priority:-2 extents:1 across:2096472k

Using TMPFS for Cache

The method here will show how to create and use a TMPFS filesystem to hold all the static files created by WP-Super Cache. These static files are served to visitors instead of loading php for every request, so by moving those static files to TMPFS your server will be able to access and start sending your site to the browser 30x faster!

The WP-Super Cache plugin stores all the static files in the wp-content/cache folder of your WordPress installation, so to enable TMPFS we simply will create a new TMPFS filesystem and mount it to the wp-content/cache folder. That makes anything in that folder (all the static files) be part of the TMPFS filesystem.

Boosting Cache with TMPFS

There are a lot of maybe new concepts surrounding TMPFS and it may seem too complicated, but the process of actually setting up a robust tmpfs to use for wp-super-cache's cache folder is actually very simple. As long as you have shell access to your server and the permissions required (any sudo or private server should be good to go) you can set this up in a couple minutes and not really have to give it a second thought or debug anything. Here's the process I've used on several client sites.

Create a TMPFS Filesystem and Mount at /wp-content/cache/
Restore TMPFS Cached Files across Reboots
Keep a semi-current mirror of the TMPFS files on Disk

Create TMPFS at wp-content/cache

/etc/fstab

tmpfs /home/askapache/wp-content/cache tmpfs defaults,size=2g,noexec,nosuid,uid=648,gid=648,mode=1755 0 0

Restoring TMPFS across Reboots

In /etc/rc.local

ionice -c3 -n7 nice -n 19 rsync -ahv --stats --delete /_b/tmpfs/cache/ /home/askapache/wp-content/cache/ 1>/dev/null

Mirroring TMPFS to Disk

Cronjob entry

*/5 * * * * /usr/bin/ionice -c3 -n7 /bin/nice -n 19 /usr/bin/rsync -ah --stats --delete /home/askapache/wp-content/cache/ /_b/tmpfs/cache/ 1>/dev/null

/tmp, /var/run, and /var/lock

The directories /tmp, /var/run, and /var/lock contain files that are not needed across reboots. This means they are ideal candidates for tmpfs. HEre's how to do it.

tmpfs /var/run tmpfs defaults,rw,nosuid,mode=0755 0 0

tmpfs /var/lock tmpfs defaults,rw,noexec,nosuid,nodev,mode=1777 0 0

Resize /dev/shm

You can view your current /dev/shm size with the command df -ha|grep /dev/shm then if you want to resize that use the command:

mount -t tmpfs -o remount,size-2G,rw,nosuid,nodev tmpfs /dev/shm

Secure /dev/shm:
 
Step 1: Edit your /etc/fstab:
 
nano -w /etc/fstab
 
Locate:
 
none /dev/shm tmpfs defaults,rw 0 0
 
Change it to:
 
none /dev/shm tmpfs defaults,nosuid,noexec,rw 0 0
 
Step 2: Remount /dev/shm:
 
mount -o remount /dev/shm
 
guilt makes extensive use of the '$$' shell variable for temporary
files in /tmp. This is a serious security vulnerability; on multi-user
systems it allows an attacker to clobber files with something like the
following:
 
for i in `seq 1 32768`; do
ln -sf /etc/passwd /tmp/guilt.log.$i;
done
 
(In this example, if root does e.g. 'guilt push', /etc/passwd will get
clobbered.)

Securing and Using /tmp

tmpfs mount parameters

A good way to find a good tmpfs upper-bound is to use top to monitor your system's swap usage during peak usage periods. Then, make sure that you specify a tmpfs upper-bound that's slightly less than the sum of all free swap and free RAM during these peak usage times.

mode=1777 sets sticky bit on directory. Only file owners can delete files in this directory.

The following parameters accept a suffix k, m or g for Ki, Mi, Gi (binary kilo, mega and giga) and can be changed on remount.

size: Override default maximum size of the filesystem. The size is given in bytes, and rounded down to entire pages. The default is half of the memory.The limit of allocated bytes for this tmpfs instance. The default is half of your physical RAM without swap. If you oversize your tmpfs instances the machine will deadlock since the OOM handler will not be able to free that memory.
nr_inodes: Set number of inodes.
nr_blocks: Set number of blocks.
mode: The permissions as an octal number
uid: The user id
gid: The group id

mount -t tmpfs -o size=10G,nr_inodes=10k,mode=700 tmpfs /mytmpfs

Will give you tmpfs instance on /mytmpfs which can allocate 10GB RAM/SWAP in 10240 inodes and it is only accessible by root.

Using tmpfs for /tmp storage

Many users find it very convenient to use tmpfs for /tmp and /var/tmp which does a number of positive things. Any temporary files are instead created in RAM not your hard-drive, which means that reading/writing/accessing those temporary files by various processes doesn't slow down your hard-drive read/writes/accesses for your other processes. This also has a side-effect of making your hard-drive have a longer life as it reduces activity by a huge amount.

Remember that tmpfs uses both RAM and swap, so make sure your machine has a large swapfile, like gigabytes. If your tmpfs consumes all the swap and RAM then you are screwed, so make sure that you correctly set the mount options for the tmpfs so that it doesn't do that. If your /tmp or /var/tmp gets filled with tmp files that for some reason don't get deleted except at reboot, and your machine has a very high uptime, then you will want to run some cron jobs to periodically clean the /tmp and /var/tmp directories of older files...

Here's an example scenario: let's say that we have an existing filesystem mounted at /tmp. However, we decide that we'd like to start using tmpfs for /tmp storage.

with recent 2.4 kernels, you can mount your new /tmp filesystem without getting the "device is busy" error:

mount tmpfs /tmp -t tmpfs -o size=64m

With a single command, your new tmpfs /tmp filesystem is mounted at /tmp, on top of the already-mounted partition, which can no longer be directly accessed. However, while you can't get to the original /tmp, any processes that still have open files on this original filesystem can continue to access them. And, if you umount your tmpfs-based /tmp, your original mounted /tmp filesystem will reappear. In fact, you can mount any number of filesystems to the same mountpoint, and the mountpoint will act like a stack; unmount the current filesystem, and the last-most-recently mounted filesystem will reappear from underneath.

Bind Mounts

Using bind mounts, we can mount all, or even part of an already-mounted filesystem to another location, and have the filesystem accessible from both mountpoints at the same time!

For example, you can use bind mounts to mount your existing /tmp filesystem to /sites/askapache.com/tmp, as follows:

mount --bind /tmp /sites/askapache.com/tmp

Now, if you look inside /sites/askapache.com/tmp, you'll see your /tmp filesystem and all its files. And if you modify a file on your /tmp filesystem, you'll see the modifications in /sites/askapache.com/tmp as well. This is because they are one and the same filesystem; the kernel is simply mapping the filesystem to two different mountpoints for us.

Note that when you mount a filesystem somewhere else, any filesystems that were mounted to mountpoints inside the bind-mounted filesystem will not be moved along. In other words, if you have /tmp/cache on a separate filesystem, the bind mount we performed above will leave /sites/askapache.com/tmp/cache empty. You'll need an additional bind mount command to allow you to browse the contents of /tmp/cache at /sites/askapache.com/tmp/cache:

mount --bind /tmp/cache /sites/askapache.com/tmp/cache

Bind mounting and /dev/shm

glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for POSIX shared memory (shm_open, shm_unlink). Adding the following line to /etc/fstab should take care of this:

tmpfs  /dev/shm  tmpfs  defaults  0 0

Many systems by default have a tmpfs filesystem mounted at /dev/shm that defaults to a size of half of your physical RAM without swap. Say you decide that you'd like to start using tmpfs for /tmp, which currently lives on your root filesystem. Rather than mounting a new tmpfs filesystem to /tmp (which is possible), you may decide that you'd like the new /tmp to share the currently mounted /dev/shm filesystem. However, while you could bind mount /dev/shm to /tmp and be done with it, your /dev/shm contains some directories that you don't want to appear in /tmp. So, what do you do? How about this:

mkdir /dev/shm/tmp
chmod 1777 /dev/shm/tmp
mount --bind /dev/shm/tmp /tmp

In this example, we first create a /dev/shm/tmp directory and then give it 1777 perms, the proper permissions for /tmp. Now that our directory is ready, we can mount /dev/shm/tmp, and only /dev/shm/tmp to /tmp. So, while /tmp/foo would map to /dev/shm/tmp/foo, there's no way for you to access the /dev/shm/bar file from /tmp.

/etc/default/tmpfs WorkAround

$ cat /etc/default/tmpfs
# SHM_SIZE sets the maximum size (in bytes) that the /dev/shm tmpfs can use.
# If this is not set then the size defaults to the value of TMPFS_SIZE
# if that is set; otherwise to the kernel's default.
#
# The size will be rounded down to a multiple of the page size, 4096 bytes.
SHM_SIZE=524288000
# TMPFS_SIZE sets the max size that /dev/shm can use.  By default, the
# kernel sets this upper limit to half of available memory.
TMPFS_SIZE=524288000

RSYNC vs. CP

rsync [options]  SRC DEST
rsync -av --delete --stats /home/wincom/public_html/wp-content/cache/ /backups/tmp-mnt/cache/
-a, --archive               archive mode; same as -rlptgoD (no -H)
-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks
-p, --perms                 preserve permissions
-t, --times                 preserve times
-g, --group                 preserve group
-o, --owner                 preserve owner (super-user only)
-D                          same as --devices --specials
    --devices               preserve device files (super-user only)
    --specials              preserve special files
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer

Mount Options

The following options apply to any file system that is being mounted (but not every file system actually honors them)

async All I/O to the file system should be done asynchronously.
atime Update inode access time for each access. This is the default.
auto Can be mounted with the -a option.
defaults Use default options: rw, suid, dev, exec, auto, nouser, and async.
dev Interpret character or block special devices on the file system.
exec Permit execution of binaries.
group Allow an ordinary (i.e., non-root) user to mount the file system if one of his groups matches the group of the device. This option implies the options nosuid and nodev (unless overridden by subsequent options, as in the option line group,dev,suid).
mand Allow mandatory locks on this filesystem. See fcntl(2).
_netdev The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).
noatime Do not update inode access times on this file system (e.g, for faster access on the news spool to speed up news servers).
nodiratime Do not update directory inode access times on this filesystem.
noauto Can only be mounted explicitly (i.e., the -a option will not cause the file system to be mounted).
nodev Do not interpret character or block special devices on the file system.
noexec Do not allow direct execution of any binaries on the mounted file system. (Until recently it was possible to run binaries anyway using a command like /lib/ld*.so /mnt/binary. This trick fails since Linux 2.4.25 / 2.6.0.)
nomand Do not allow mandatory locks on this filesystem.
nosuid Do not allow set-user-identifier or set-group-identifier bits to take effect. (This seems safe, but is in fact rather unsafe if you have suidperl(1) installed.)
nouser Forbid an ordinary (i.e., non-root) user to mount the file system. This is the default.
owner Allow an ordinary (i.e., non-root) user to mount the file system if he is the owner of the device. This option implies the options nosuid and nodev (unless overridden by subsequent options, as in the option line owner,dev,suid).
remount Attempt to remount an already-mounted file system. This is commonly used to change the mount flags for a file system, especially to make a readonly file system writeable. It does not change device or mount point.
ro Mount the file system read-only.
_rnetdev Like _netdev, except "fsck -a" checks this filesystem during rc.sysinit.
rw Mount the file system read-write.
suid Allow set-user-identifier or set-group-identifier bits to take effect.
sync All I/O to the file system should be done synchronously. In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.
dirsync All directory updates within the file system should be done synchronously. This affects the following system calls: creat, link, unlink, symlink, mkdir, rmdir, mknod and rename.
user Allow an ordinary user to mount the file system. The name of the mounting user is written to mtab so that he can unmount the file system again. This option implies the options noexec, nosuid, and nodev (unless overridden by subsequent options, as in the option line user,exec,dev,suid).
users Allow every user to mount and unmount the file system. This option implies the options noexec, nosuid, and nodev (unless overridden by subsequent options, as in the option line users,exec,dev,suid).

Filesystems

You can find out what is filesystems are in place by using one of the following linux commands:

cat /etc/fstab
cat /etc/mtab
cat /proc/mounts
df -a

/etc/fstab

       /etc/fstab        file system table
       /etc/mtab         table of mounted file systems
       /etc/mtab~        lock file
       /etc/mtab.tmp     temporary file
       /etc/filesystems  a list of filesystem types to try

From /etc/mtab

none /tmp tmpfs size=128m,mode=1777 0 0

From /proc/mounts

none /tmp tmpfs rw,nodev,relatime,size=131072k 0 0

/etc/fstab

It is possible that files /etc/mtab and /proc/mounts don’t match. The first file is based only on the mount command options, but the content of the second file also depends on the kernel and others settings (e.g. remote NFS server. In particular case the mount command may reports unreliable information about a NFS mount point and the /proc/mounts file usually contains more reliable information.)

This file is used in three ways:

The following command (usually given in a bootscript) causes all file systems mentioned in fstab (of the proper type and/or having or not having the proper options) to be mounted as indicated, except for those whose line contains the noauto keyword. Adding the -F option will make mount fork, so that the filesystems are mounted simultaneously.
```
mount -a [-t type] [-O optlist]
```
When mounting a file system mentioned in fstab, it suffices to give only the device, or only the mount point.
Normally, only the superuser can mount file systems. However, when fstab contains the user option on a line, anybody can mount the corresponding system.

The programs mount and umount maintain a list of currently mounted file systems in the file /etc/mtab.

Only the user that mounted a filesystem can unmount it again. If any user should be able to unmount, then use users instead of user in the fstab line. The owner option is similar to the user option, with the restriction that the user must be the owner of the special file. The group option is similar, with the restriction that the user must be member of the group of the special file.

The order of records in fstab is important because fsck(8), mount(8), and umount(8) sequentially iterate through fstab doing their thing.

The first field, (fs_spec)

Describes the block special device or remote filesystem to be mounted. For ordinary mounts it will hold (a link to) a block special device node (as created by mknod(8)) for the device to be mounted, like ‘/dev/cdrom’ or ‘/dev/sdb7’. For NFS mounts one will have :

, e.g., ‘knuth.aeb.nl:/’. For procfs, use ‘proc’.

Instead of giving the device explicitly, one may indicate the (ext2 or xfs) filesystem that is to be mounted by its UUID or volume label (cf. e2label(8) or xfs_admin(8)), writing LABEL= or UUID=, e.g., ‘LABEL=Boot’ or ‘UUID=3e6be9de-8139-11d1-9106-a43f08d823a6’. This will make the system more robust: adding or removing a SCSI disk changes the disk device name but not the filesystem volume label.

The second field, (fs_file)

Describes the mount point for the filesystem. For swap partitions, this field should be specified as ‘none’. If the name of the mount point contains spaces these can be escaped as ‘\040’.

The third field, (fs_vfstype), describes the type of the filesystem. Linux supports lots of filesystem types, such as adfs, affs, autofs, coda, coherent, cramfs, devpts, efs, ext2, ext3, hfs, hpfs, iso9660, jfs, minix, msdos, ncpfs, nfs, ntfs, proc, qnx4, reiserfs, romfs, smbfs, sysv, tmpfs, udf, ufs, umsdos, vfat, xenix, xfs, and possibly others. For more details, see mount(8). For the filesystems currently supported by the running kernel, see /proc/filesystems. An entry swap denotes a file or partition to be used for swapping, cf. swapon(8). An entry ignore causes the line to be ignored. This is useful to show disk partitions which are currently unused.

The fourth field, (fs_mntops)

Describes the mount options associated with the filesystem. It is formatted as a comma separated list of options. It contains at least the type of mount plus any additional options appropriate to the filesystem type. For documentation on the available options for non-nfs file systems, see mount(8). For documentation on all nfs-specific options have a look at nfs(5).

Common for all types of file system are the options:

noauto: (do not mount when "mount -a" is given, e.g., at boot time)
user: (allow a user to mount)
owner: (allow device owner to mount)
pamconsole: (allow a user at the console to mount)
comment: (e.g., for use by fstab-maintaining programs).

The fifth field, (fs_freq)

Used for these filesystems by the dump(8) command to determine which filesystems need to be dumped. If the fifth field is not present, a value of zero is returned and dump will assume that the filesystem does not need to be dumped.

The sixth field, (fs_passno)

Used by the fsck(8) program to determine the order in which filesystem checks are done at reboot time. The root filesystem should be specified with a fs_passno of 1, and other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. If the sixth field is not present or zero, a value of zero is returned and fsck will assume that the filesystem does not need to be checked.

Use tmpfs for MySQL

--tmpdir=path, -t path

The path of the directory to use for creating temporary files. It might be useful if your default /tmp directory resides on a partition that is too small to hold temporary tables. Starting from MySQL 4.1.0, this option accepts several paths that are used in round-robin fashion. Paths should be separated by colon characters (“:”) on Unix and semicolon characters (“;”) on Windows, NetWare, and OS/2. If the MySQL server is acting as a replication slave, you should not set --tmpdir to point to a directory on a memory-based file system or to a directory that is cleared when the server host restarts. For more information about the storage location of temporary files, see Section A.1.4.4, “Where MySQL Stores Temporary Files”. A replication slave needs some of its temporary files to survive a machine restart so that it can replicate temporary tables or LOAD DATA INFILE operations. If files in the temporary file directory are lost when the server restarts, replication fails.

On Unix, MySQL uses the value of the TMPDIR environment variable as the path name of the directory in which to store temporary files. If TMPDIR is not set, MySQL uses the system default, which is usually /tmp, /var/tmp, or /usr/tmp. If the file system containing your temporary file directory is too small, you can use the --tmpdir option to mysqld to specify a directory in a file system where you have enough space. Starting from MySQL 4.1, the --tmpdir option can be set to a list of several paths that are used in round-robin fashion. Paths should be separated by colon characters (“:”) on Unix and semicolon characters (“;”) on Windows, NetWare, and OS/2. Note To spread the load effectively, these paths should be located on different physical disks, not different partitions of the same disk. If the MySQL server is acting as a replication slave, you should not set --tmpdir to point to a directory on a memory-based file system or to a directory that is cleared when the server host restarts. A replication slave needs some of its temporary files to survive a machine restart so that it can replicate temporary tables or LOAD DATA INFILE operations. If files in the temporary file directory are lost when the server restarts, replication fails. MySQL creates all temporary files as hidden files. This ensures that the temporary files are removed if mysqld is terminated. The disadvantage of using hidden files is that you do not see a big temporary file that fills up the file system in which the temporary file directory is located.

Shell Script for Firefox tmpfs

#!/bin/bash
### Bind temporary directories to /dev/shm ###
# I do this instead of mounting tmpfs on the #
# directories, so less memory gets wasted.   #
##############################################
mkdir /dev/shm/{tmp,lock}
mount --bind /dev/shm/tmp /tmp
mount --bind /dev/shm/tmp /var/tmp
mount --bind /dev/shm/lock /var/lock
chmod 1777 /dev/shm/{tmp,lock}

Hey! You made it!@ at least to the bottom of the page.. I still have to finish this article, so check back in a few months.

30x Faster Cache and Site Speed with TMPFS originally appeared on AskApache.com

COMPUTER SECURITY TOOLBOX

AskApache — Sat, 27 Sep 2008 00:29:52 +0000

AskApache.com

List of mainly obscure security software geared more for the master pentester. These are mostly for unix, bsd, and mac and many are difficult to install and setup (require custom servers, inside access points, obscure libraries). Only programs that output data are included, so no actual exploits or anything. Most of these output extremely useful albeit extremely technical information.

You may be looking for the article: Vulnerability Scanners Review, or Top 5 Vulnerability Port Scanners

Obscure/Rare Security Software

rwhois: really great addition to using whois. Get additional info not on whois, query rwhois servers.
lft: useful alternative method of tracerouteing. oppleman
packit: define (spoof) nearly all TCP, UDP, ICMP, IP, ARP, RARP, and Ethernet header options
etherape: really cool graphical program that displays connections and protocols similar to cheops.
amap: fingerprinting
xprobe2: fingerprinting
p0f2: really exceptional fingerprinting. can be passively run in the BG.
firewalk: good packetfiltering enumerator
BGPview: bgp anyone?
icmpenum: icmp fingerprinting
dnstracer: awesome and creative graphical output of dns
ssldump: not really that useful but impressive in a report
ftester: for master pentesters only — get the lowdown on your packetfiltering
mtr: alternative traceroute
MRTG: favorite tool of ISPs, many uses here
host: don't forget this one
ike-scan: scan for vpns
upnpscan: scan for upnp devices
ftp-spider: get info on ftp server
traceproto: very nice alternative to traceroute/firewalk
sing: packet crafting
nmbscan: NBM Scanner
nbtscan: NBT Scanner
admsmb: ADMsmb
netleak: Netleak
dmitry
sara: Original security auditing software
isic: ISIC
dnsa: DNS
nemesis: Packet Crafting
zodiacdns: DNS Hacking
fragroute: Fragmented Packet Crafter/Scanner
sentry 2.0
Caecus
C-Parse
ftester: Master Pentesting Tool, Map out the filtering of your firewall with internal and external nodes
pchar

More common security programs

Nessus: Premier UNIX vulnerability assessment tool - Nessus is the best free network vulnerability scanner available, and the best to run on UNIX at any price. It is constantly updated, with more than 11,000 plugins for the free (but registration and EULA-acceptance required) feed. Key features include remote and local (authenticated) security checks, a client/server architecture with a GTK graphical interface, and an embedded scripting language for writing your own plugins or understanding the existing ones.
Wireshark: Sniffing the glue that holds the Internet together - Wireshark (known as Ethereal until a trademark dispute in Summer 2006) is a fantastic open source network protocol analyzer for Unix and Windows. It allows you to examine data from a live network or from a capture file on disk. You can interactively browse the capture data, delving down into just the level of packet detail you need. Wireshark has several powerful features, including a rich display filter language and the ability to view the reconstructed stream of a TCP session. It also supports hundreds of protocols and media types. A tcpdump-like console version named tethereal is included. One word of caution is that Ethereal has suffered from dozens of remotely exploi security holes, so stay up-to-date and be wary of running it on unusted or hostile networks (such as security conferences).
Snort: A Everyone's favorite open source IDS - This lightweight network inusion detection and prevention system excels at traffic analysis and packet logging on IP networks. Through protocol analysis, content searching, and various pre-processors, Snort detects thousands of worms, vulnerability exploit attempts, port scans, and other suspicious behavior. Snort uses a flexible rule-based language to describe traffic that it should collect or pass, and a modular detection engine. Also check out the free Basic Analysis and Security Engine (BASE), a web interface for analyzing Snort alerts. Open source Snort works fine for many individuals, small businesses, and departments. Parent company SourceFire offers a complimentary product line with more enterprise-level features and real-time rule updates. They offer a free (with registration) 5-day-delayed rules feed, and you can also find many great free rules at Bleeding Edge Snort.
Netcat: The network Swiss army knife - This simple utility reads and writes data across TCP or UDP network connections. It is designed to be a reliable back-end tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need, including port binding to accept incoming connections. The original Netcat was released by Hobbit in 1995, but it hasn't been maintained despite its immense popularity. It can sometimes even be hard to find nc110.tgz. The flexibility and usefulness of this tool have prompted people to write numerous other Netcat implementations - often with modern features not found in the original. One of the most interesting is Socat, which extends Netcat to support many other socket types, SSL encryption, SOCKS proxies, and more. It even made this list on its own merits. There is also Chris Gibson's Ncat, which offers even more features while remaining por and compact. Other takes on Netcat include OpenBSD's nc, Cryptcat, Netcat6, PNetcat, SBD, and so-called GNU Netcat.
Metasploit Framework: Hack the Planet - Metasploit took the security world by storm when it was released in 2004. No other new tool even broke into the top 15 of this list, yet Metasploit comes in at #5, ahead of many well-loved tools that have been developed for more than a decade. It is an advanced open-source platform for developing, testing, and using exploit code. The extensible model through which payloads, encoders, no-op generators, and exploits can be integrated has made it possible to use the Metasploit Framework as an outlet for cutting-edge exploitation research. It ships with hundreds of exploits, as you can see in their online exploit building demo. This makes writing your own exploits easier, and it certainly beats scouring the darkest corners of the Internet for illicit shellcode of dubious quality. Similar professional exploitation tools, such as Core Impact and Canvas already existed for wealthy users on all sides of the ethical specum. Metasploit simply brought this capability to the masses.
Hping2: A network probing utility like ping on steroids - This handy little utility assembles and sends custom ICMP, UDP, or TCP packets and then displays any replies. It was inspired by the ping command, but offers far more conol over the probes sent. It also has a handy aceroute mode and supports IP fragmentation. This tool is particularly useful when ying to aceroute/ping/probe hosts behind a firewall that blocks attempts using the standard utilities. This often allows you to map out firewall rulesets. It is also great for learning more about TCP/IP and experimenting with IP protocols.
Kismet: A powerful wireless sniffer - Kismet is an console (ncurses) based 802.11 layer2 wireless network detector, sniffer, and inusion detection system. It identifies networks by passively sniffing (as opposed to more active tools such as NetStumbler), and can even decloak hidden (non-beaconing) networks if they are in use. It can automatically detect network IP blocks by sniffing TCP, UDP, ARP, and DHCP packets, log traffic in Wireshark/TCPDump compatible format, and even plot detected networks and estimated ranges on downloaded maps. As you might expect, this tool is commonly used for wardriving. Oh, and also warwalking, warflying, and warskating
Tcpdump: The classic sniffer for network monitoring and data acquisition - Tcpdump is the IP sniffer we all used before Ethereal (Wireshark) came on the scene, and many of us continue to use it frequently. It may not have the bells and whistles (such as a pretty GUI or parsing logic for hundreds of application protocols) that Wireshark has, but it does the job well and with fewer security holes. It also requires fewer system resources. While it doesn't receive new features often, it is actively maintained to fix bugs and portability problems. It is great for acking down network problems or monitoring activity. There is a separate Windows port named WinDump. TCPDump is the source of the Libpcap/WinPcap packet capture library, which is used by Nmap among many other tools.
Cain and Abel: The top password recovery tool for Windows - UNIX users often smugly assert that the best free security tools support their platform first, and Windows ports are often an afterthought. They are usually right, but Cain & Abel is a glaring exception. This Windows-only password recovery tool handles an enormous variety of tasks. It can recover passwords by sniffing the network, cracking encrypted passwords using Dictionary, Brute-Force and Cryptanalysis attacks, recording VoIP conversations, decoding scrambled passwords, revealing password boxes, uncovering cached passwords and analyzing routing protocols. It is also well documented.
John the Ripper: A powerful, flexible, and fast multi-platform password hash cracker - John the Ripper is a fast password cracker, currently available for many flavors of Unix (11 are officially supported, not counting different architectures), DOS, Win32, BeOS, and OpenVMS. Its primary purpose is to detect weak Unix passwords. It supports several crypt(3) password hash types which are most commonly found on various Unix flavors, as well as Kerberos AFS and Windows NT/2000/XP LM hashes. Several other hash types are added with conibuted patches. You will want to start with some wordlists, which you can find here, here, or here.
Ettercap: In case you still thought switched LANs provide much exa security - Ettercap is a terminal-based network sniffer/interceptor/logger for ethernet LANs. It supports active and passive dissection of many protocols (even ciphered ones, like ssh and https). Data injection in an established connection and filtering on the fly is also possible, keeping the connection synchronized. Many sniffing modes were implemented to give you a powerful and complete sniffing suite. Plugins are supported. It has the ability to check whether you are in a switched LAN or not, and to use OS fingerprints (active or passive) to let you know the geomey of the LAN.
Nikto: A more comprehensive web scanner - Nikto is an open source (GPL) web server scanner which performs comprehensive tests against web servers for multiple items, including over 3200 potentially dangerous files/CGIs, versions on over 625 servers, and version specific problems on over 230 servers. Scan items and plugins are frequently updated and can be automatically updated (if desired). It uses Whisker/libwhisker for much of its underlying functionality. It is a great tool, but the value is limited by its infrequent updates. The newest and most critical vulnerabilities are often not detected.
Ping/telnet/dig/aceroute/whois/netsta: The basics - While there are many whiz-bang high-tech tools out there to assist in security auditing, don't forget about the basics! Everyone should be very familiar with these tools as they come with most operating systems (except that Windows omits whois and uses the name acert). They can be very handy in a pinch, although for more advanced usage you may be better off with Hping2 and Netcat.
OpenSSH / PuTTY / SSH: A secure way to access remote computers - SSH (Secure Shell) is the now ubiquitous program for logging into or executing commands on a remote machine. It provides secure encrypted communications between two unusted hosts over an insecure network, replacing the hideously insecure telnet/rlogin/rsh alternatives. Most UNIX users run the open source OpenSSH server and client. Windows users often prefer the free PuTTY client, which is also available for many mobile devices. Other Windows users prefer the nice terminal-based port of OpenSSH that comes with Cygwin. Dozens of other free and proprietary clients exist. You can explore them here or here.
THC Hydra: A Fast network authentication cracker which support many different services - When you need to brute force crack a remote authentication service, Hydra is often the tool of choice. It can perform rapid dictionary attacks against more then 30 protocols, including telnet, ftp, http, https, smb, several databases, and much more. Like THC Amap this release is from the fine folks at THC.
Paros proxy: A web application vulnerability assessment proxy - A Java based web proxy for assessing web application vulnerability. It supports editing/viewing HTTP/HTTPS messages on-the-fly to change items such as cookies and form fields. It includes a web traffic recorder, web spider, hash calculator, and a scanner for testing common web application attacks such as SQL injection and cross-site scripting.
Dsniff: A suite of powerful network auditing and peneation-testing tools - This popular and well-engineered suite by Dug Song includes many tools. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected ssh and https sessions by exploiting weak bindings in ad-hoc PKI. A separately maintained partial Windows port is available here. Overall, this is a great toolset. It handles pretty much all of your password sniffing needs.
NetStumbler: Free Windows 802.11 Sniffer - Netstumbler is the best known Windows tool for finding open wireless access points ("wardriving"). They also disibute a WinCE version for PDAs and such named Ministumbler. The tool is currently free but Windows-only and no source code is provided. It uses a more active approach to finding WAPs than passive sniffers such as Kismet or KisMAC.
THC Amap: An application fingerprinting scanner - Amap is a great tool for determining what application is listening on a given port. Their database isn't as large as what Nmap uses for its version detection feature, but it is definitely worth ying for a 2nd opinion or if Nmap fails to detect a service. Amap even knows how to parse Nmap output files. This is yet another valuable tool from the great guys at THC.
GFI LANguard: A commercial network security scanner for Windows - GFI LANguard scans IP networks to detect what machines are running. Then it ies to discern the host OS and what applications are running. I also ies to collect Windows machine's service pack level, missing security patches, wireless access points, USB devices, open shares, open ports, services/applications active on the computer, key registry enies, weak passwords, users and groups, and more. Scan results are saved to an HTML report, which can be customized/queried. It also includes a patch manager which detects and installs missing patches. A free ial version is available, though it only works for up to 30 days.
Aircrack: The fastest available WEP/WPA cracking tool - Aircrack is a suite of tools for 802.11a/b/g WEP and WPA cracking. It can recover a 40 through 512-bit WEP key once enough encrypted packets have been gathered. It can also attack WPA 1 or 2 networks using advanced cryptographic methods or by brute force. The suite includes airodump (an 802.11 packet capture program), aireplay (an 802.11 packet injection program), aircrack (static WEP and WPA-PSK cracking), and airdecap (decrypts WEP/WPA capture files).
Superscan: A Windows-only port scanner, pinger, and resolver - SuperScan is a free Windows-only closed-source TCP/UDP port scanner by Foundstone. It includes a variety of additional networking tools such as ping, aceroute, http head, and whois.
Netfilter: The current Linux kernel packet filter/firewall - Netfilter is a powerful packet filter implemented in the standard Linux kernel. The userspace ips tool is used for configuration. It now supports packet filtering (stateless or stateful), all kinds of network address and port anslation (NAT/NAPT), and multiple API layers for 3rd party extensions. It includes many different modules for handling unruly protocols such as FTP. For other UNIX platforms, see Openbsd PF (OpenBSD specific), or IP Filter. Many personal firewalls are available for Windows (Tiny,Zone Alarm, Norton, Kerio, ...), though none made this list. Microsoft included a very basic firewall in Windows XP SP2, and will nag you incessantly until you install it.
Retina: Commercial vulnerability assessment scanner by eEye - Like Nessus, Retina's function is to scan all the hosts on a network and report on any vulnerabilities found. It was written by eEye, who are well known for their security research.
Angry IP Scanner: A fast windows IP scanner and port scanner - Angry IP Scanner can perform basic host discovery and port scans on Windows. Its binary file size is very small compared to other scanners and other pieces of information about the target hosts can be extended with a few plugins.
RKHunter: An Unix Rootkit Detector - RKHunter is scanning tool that checks for signs of various pieces of nasty software on your system like rootkits, backdoors and local exploits. It runs many tests, including MD5 hash comparisons, default filenames used by rootkits, wrong file permissions for binaries, and suspicious sings in LKM and KLD modules.
Ike-scan: VPN detector/scanner - Ike-scan exploits ansport characteristics in the Internet Key Exchange (IKE) service, the mechanism used by VPNs to establish a connection between a server and a remote client. It scans IP addresses for VPN servers by sending a specially crafted IKE packet to each host within a network. Most hosts running IKE will respond, identifying their presence. The tool then remains silent and monitors reansmission packets. These reansmission responses are recorded, displayed and matched against a known set of VPN product fingerprints. Ike-scan can VPNs from manufacturers including Checkpoint, Cisco, Microsoft, Nortel, and Watchguard.
Arpwatch: Keeps ack of ethernet/IP address pairings and can detect certain monkey business Arpwatch is the classic ARP man-in-the-middle attack detector from LBNL's Network Research Group. It syslogs activity and reports certain changes via email. Arpwatch uses LibPcap to listen for ARP packets on a local ethernet interface.
KisMAC: A A GUI passive wireless stumbler for Mac OS X - This popular stumbler for Mac OS X offers many of the features of its namesake Kismet, though the codebase is entirely different. Unlike console-based Kismet, KisMAC offers a pretty GUI and was around before Kismet was ported to OS X. It also offers mapping, Pcap-format import and logging, and even some decryption and deauthentication attacks.
OSSEC HIDS: An Open Source Host-based Inusion Detection System - OSSEC HIDS performs log analysis, integrity checking, rootkit detection, time-based alerting and active response. In addition to its IDS functionality, it is commonly used as a SEM/SIM solution. Because of its powerful log analysis engine, ISPs, universities and data centers are running OSSEC HIDS to monitor and analyze their firewalls, IDSs, web servers and authentication logs.
Openbsd PF: The OpenBSD Packet Filter - Like Netfilter and IP Filter on other platforms, OpenBSD users love PF, their firewall tool. It handles network address anslation, normalizing TCP/IP traffic, providing bandwidth conol, and packet prioritization. It also offers some eccenic features, such as passive OS detection. Coming from the same guys who created OpenBSD, you can ust that it has been well audited and coded to avoid the sort of security holes we have seen in other packet filters.
Nemesis: Packet injection simplified - The Nemesis Project is designed to be a commandline-based, por human IP stack for UNIX/Linux (and now Windows!). The suite is broken down by protocol, and should allow for useful scripting of injected packet streams from simple shell scripts. If you enjoy Nemesis, you might also want to look at Hping2 as they complement each other well.
Tor: An anonymous Internet communication system - Tor is a toolset for a wide range of organizations and people that want to improve their safety and security on the Internet. Using Tor can help you anonymize web browsing and publishing, instant messaging, irc, ssh, and other applications that use the TCP protocol. Tor also provides a platform on which software developers can build new applications with built-in anonymity, safety, and privacy features. For a free cross-platform GUI, users recommend Vidalia
Knoppix: A general-purpose boo live system on CD or DVD - Knoppix consists of a representative collection of GNU/Linux software, automatic hardware detection, and support for many graphics cards, sound cards, SCSI and USB devices and other peripherals. KNOPPIX can be used as a productive Linux system for the desktop, educational CD, rescue system, or as many nmap survey takers attest, a por security tool. For a security-specific Linux disibution see Backack.
ISS Internet Scanner: Application-level vulnerability assessment - Internet Scanner started off in '92 as a tiny open source scanner by Christopher Klaus. Now he has grown ISS into a billion-dollar company with a myriad of security products.
Fport: Foundstone's enhanced netstat - Fport reports all open TCP/IP and UDP ports on the machine you run it on and shows what application opened each port. So it can be used to quickly identify unknown open ports and their associated applications. It only runs on Windows, but many UNIX systems now provided this information via netstat (y 'netstat -pan' on Linux). Here is a PDF-Format SANS article on using Fport and analyzing the results.
chkrootkit: Locally checks for signs of a rootkit - chkrootkit is a flexible, por tool that can check for many signs of rootkit inusion on Unix-based systems. Its features include detecting binary modification, utmp/wtmp/lastlog modifications, promiscuous interfaces, and malicious kernel modules.
SPIKE Proxy: HTTP Hacking - Spike Proxy is an open source HTTP proxy for finding security flaws in web sites. It is part of the Spike Application Testing Suite and supports automated SQL injection detection, web site crawling, login form brute forcing, overflow detection, and directory aversal detection.
OpenBSD: The Proactively Secure Operating System - OpenBSD is one of the only operating systems to eat security as their very highest priority. Even higher than usability in some cases. But their enviable security record speaks for itself. They also focus on stability and fight to obtain documentation for the hardware they wish to support. Perhaps their greatest achievement was creating OpenSSH. OpenBSD users also love [pf], their firewall tool.
Yersinia: A multi-protocol low-level attack tool - Yersinia is a low-level protocol attack tool useful for peneation testing. It is capable of many diverse attacks over multiple protocols, such as becoming the root role in the Spanning ee (Spanning ee Protocol), creating virtual CDP (Cisco Discovery Protocol) neighbors, becoming the active router in a HSRP (Hot Standby Router Protocol) scenario, faking DHCP replies, and other low-level attacks.
Nagios: An open source host, service and network monitoring program - Nagios is a system and network monitoring application. It watches hosts and services that you specify, alerting you when things go bad and when they get better. Some of its many features include monitoring of network services (smtp, pop3, http, nntp, ping, etc.), monitoring of host resources (processor load, disk usage, etc.), and contact notifications when service or host problems occur and get resolved (via email, pager, or user-defined method).
Fragroute / Fragrouter: A network inusion detection evasion toolkit - Fragrouter is a one-way fragmenting router - IP packets get sent from the attacker to the Fragrouter, which ansforms them into a fragmented data stream to forward to the victim. Many network IDS are unable or simply don't bother to reconsuct a coherent view of the network data (via IP fragmentation and TCP stream reassembly), as discussed in this classic paper. Fragrouter helps an attacker launch IP-based attacks while avoiding detection. It is part of the NIDSbench suite of tools by Dug Song. Fragroute is a similar tool which is also by Dug Song.
X-scan: A general scanner for scanning network vulnerabilities - A multi-threaded, plug-in-supported vulnerability scanner. X-Scan includes many features, including full NASL support, detecting service types, remote OS type/version detection, weak user/password pairs, and more. You may be able to find newer versions available here if you can deal with most of the page being written in Chinese.
Whisker/libwhisker: Rain.Forest.Puppy's CGI vulnerability scanner and library - Libwhisker is a Perl module geared geared towards HTTP testing. It provides functions for testing HTTP servers for many known security holes, particularly the presence of dangerous CGIs. Whisker is a scanner that used libwhisker but is now deprecated in favor of Nikto which also uses libwhisker.
Socat: A relay for bidirectional data ansfer - A utility similar to the venerable Netcat that works over a number of protocols and through a files, pipes, devices (terminal or modem, etc.), sockets (Unix, IP4, IP6 - raw, UDP, TCP), a client for SOCKS4, proxy CONNECT, or SSL, etc. It provides forking, logging, and dumping, different modes for interprocess communication, and many more options. It can be used, for example, as a TCP relay (one-shot or daemon), as a daemon-based socksifier, as a shell interface to Unix sockets, as an IP6 relay, for redirecting TCP-oriented programs to a serial line, or to establish a relatively secure environment (su and chroot) for running client or server shell scripts with network connections.
Sara: Security Auditor's Research Assistant - SARA is a vulnerability assessment tool that was derived from the infamous SATAN scanner. They y to release updates twice a month and y to leverage other software created by the open source community (such as Nmap and Samba).
QualysGuard: A web-based vulnerability scanner - Delivered as a service over the Web, QualysGuard eliminates the burden of deploying, maintaining, and updating vulnerability management software or implementing ad-hoc security applications. Clients securely access QualysGuard through an easy-to-use Web interface. QualysGuard features 5,000+ unique vulnerability checks, an Inference-based scanning engine, and automated daily updates to the QualysGuard vulnerability KnowledgeBase.
ClamAV: A GPL anti-virus toolkit for UNIX - ClamAV is a powerful AntiVirus scanner focused towards integration with mail servers for attachment scanning. It provides a flexible and scalable multi-threaded daemon, a command line scanner, and a tool for automatic updating via the Internet. Clam AntiVirus is based on a shared library disibuted with the Clam AntiVirus package, which you can use with your own software. Most importantly, the virus database is kept up to date.
Burpsuite: An integrated platform for attacking web applications - Burp suite allows an attacker to combine manual and automated techniques to enumerate, analyze, attack and exploit web applications. The various burp tools work together effectively to share information and allow findings identified within one tool to form the basis of an attack using another.
Brutus: A network brute-force authentication cracker - This Windows-only cracker bangs against network services of remote systems ying to guess passwords by using a dictionary and permutations thereof. It supports HTTP, POP3, FTP, SMB, TELNET, IMAP, NTP, and more. No source code is available. UNIX users should take a look at THC Hydra.
Unicornscan: Not your mother's port scanner - Unicornscan is an attempt at a User-land Disibuted TCP/IP stack for information gathering and correlation. It is intended to provide a researcher a superior interface for inoducing a stimulus into and measuring a response from a TCP/IP enabled device or network. Some of its features include asynchronous stateless TCP scanning with all variations of TCP flags, asynchronous stateless TCP banner grabbing, and active/passive remote OS, application, and component identification by analyzing responses. Like Scanrand, it isn't for the faint of heart.
Stunnel: A general-purpose SSL cryptographic wrapper - The stunnel program is designed to work as an SSL encryption wrapper between remote client and local (ine-star) or remote server. It can be used to add SSL functionality to commonly used ine daemons like POP2, POP3, and IMAP servers without any changes in the programs' code. It will negotiate an SSL connection using the OpenSSL or SSLeay libraries.
Honeyd: Your own personal honeynet Honeyd is a small daemon that creates virtual hosts on a network. The hosts can be configured to run arbiary services, and their TCP personality can be adapted so that they appear to be running certain versions of operating systems. Honeyd enables a single host to claim multiple addresses on a LAN for network simulation. It is possible to ping the virtual machines, or to aceroute them. Any type of service on the virtual machine can be simulated according to a simple configuration file. It is also possible to proxy services to another machine rather than simulating them. It has many library dependencies, which can make compiling/installing Honeyd difficult.
Fping: A parallel ping scanning program - fping is a ping(1) like program which uses the Internet Conol Message Protocol (ICMP) echo request to determine if a host is up. fping is different from ping in that you can specify any number of hosts on the command line, or specify a file containing the lists of hosts to ping. Instead of ying one host until it timeouts or replies, fping will send out a ping packet and move on to the next host in a round-robin fashion. If a host replies, it is noted and removed from the list of hosts to check. If a host does not respond within a certain time limit and/or rey limit it will be considered unreachable.
BASE: The Basic Analysis and Security Engine - BASE is a PHP-based analysis engine to search and process a database of security events generated by various IDSs, firewalls, and network monitoring tools. Its features include a query-builder and search interface for finding alerts matching different patterns, a packet viewer/decoder, and charts and statistics based on time, sensor, signature, protocol, IP address, etc.
Argus: A generic IP network ansaction auditing tool - Argus is a fixed-model Real Time Flow Monitor designed to ack and report on the status and performance of all network ansactions seen in a data network traffic stream. Argus provides a common data format for reporting flow meics such as connectivity, capacity, demand, loss, delay, and jitter on a per ansaction basis. The record format that Argus uses is flexible and extensible, supporting generic flow identifiers and meics, as well as application/protocol specific information.
Wikto: Web Server Assessment Tool - Wikto is a tool that checks for flaws in webservers. It provides much the same functionality as Nikto but adds various interesting pieces of functionality, such as a Back-End miner and close Google integration. Wikto is written for the MS .NET environment and registration is required to download the binary and/or source code.
Sguil: The Analyst Console for Network Security Monitoring - Sguil (pronounced sgweel) is built by network security analysts for network security analysts. Sguil's main component is an intuitive GUI that provides realtime events from Snort/barnyard. It also includes other components which facilitate the practice of Network Security Monitoring and event driven analysis of IDS alerts.
Scanrand: An unusually fast stateless network service and topology discovery system - Scanrand is a stateless host-discovery and port-scanner similar in design to Unicornscan. It ades off reliability for amazingly fast speeds and uses cryptographic techniques to prevent attackers from manipulating scan results. This utility is a part of a software package called Paketto Keiretsu which was written by Dan Kaminsky.
IP Filter: Por UNIX Packet Filter - IP Filter is a software package that can be used to provide network address anslation (NAT) or firewall services. It can either be used as a loadable kernel module or incorporated into your UNIX kernel; use as a loadable kernel module where possible is highly recommended. Scripts are provided to install and patch system files, as required. IP Filter is disibuted with FreeBSD, NetBSD, and Solaris. OpenBSD users should see Openbsd PF and Linux users Netfilter.
Canvas: A Comprehensive Exploitation Framework - Canvas is a commercial vulnerability exploitation tool from Dave Aitel's ImmunitySec. It includes more than 150 exploits and is less expensive than Core Impact, though it still costs thousands of dollars. You can also buy the optional VisualSploit Plugin for drag and drop GUI exploit creation. Zero-day exploits can occasionally be found within Canvas.
VMware: Multi-platform Virtualization Software - VMware virtualization software lets you run one operating system within another. This is quite useful for security researchers who commonly need to test code, exploits, etc on multiple platforms. It only runs on Windows and Linux as the host OS, but pretty much any x86 OS will run inside the virtualized environment. It is also useful for setting up sandboxes. You can browse from within a VMware window so the even if you are infected with malware, it cannot reach your host OS. And recovering the guest OS is as simple as loading a "snapshot" from prior to the infection. VMware player (executes, but can't create OS images) and VMWare Server (partitions a physical server machine into multiple virtual machines) were recently released for free. Another interesting virtualization system (Linux focused) is Xen.
Tcpaceroute: A aceroute implementation using TCP packets - The problem is that with the widespread use of firewalls on the modern Internet, many of the packets that the conventional aceroute(8) sends out (ICMP echo or UDP) end up being filtered, making it impossible to completely ace the path to the destination. However, in many cases, these firewalls will permit inbound TCP packets to specific ports that hosts sitting behind the firewall are listening for connections on. By sending out TCP SYN packets instead of UDP or ICMP ECHO packets, tcpaceroute is able to bypass the most common firewall filters.
SAINT: Security Adminisator's Integrated Network Tool - SAINT is another commercial vulnerability assessment tool (like Nessus, ISS Internet Scanner, or Retina). It runs on UNIX and used to be free and open source, but is now a commercial product.
OpenVPN: A full-featured SSL VPN solution - OpenVPN is an open-source SSL VPN package which can accommodate a wide range of configurations, including remote access, site-to-site VPNs, WiFi security, and enterprise-scale remote access solutions with load balancing, failover, and fine-grained access-conols. OpenVPN implements OSI layer 2 or 3 secure network extension using the indusy standard SSL/TLS protocol, supports flexible client authentication methods based on certificates, smart cards, and/or 2-factor authentication, and allows user or group-specific access conol policies using firewall rules applied to the VPN virtual interface. OpenVPN uses OpenSSL as its primary cryptographic library.
OllyDbg: An assembly level Windows debugger - OllyDbg is a 32-bit assembler level analyzing debugger for Microsoft Windows. Emphasis on binary code analysis makes it particularly useful in cases where source is unavailable. OllyDbg features an intuitive user interface, advanced code analysis capable of recognizing procedures, loops, API calls, switches, s, constants and sings, an ability to attach to a running program, and good multi-thread support. OllyDbg is free to download and use but no source code is provided.
Helix: A Linux Disibution with Computer Forensics in Mind - Helix is a customized disibution of the Knoppix Live Linux CD. Helix is more than just a boo live CD. You can still boot into a customized Linux environment that includes customized Linux kernels, excellent hardware detection and many applications dedicated to Incident Response and Forensics. Helix has been designed very carefully to NOT touch the host computer in any way and it is forensically sound. Helix will not auto mount swap space, or auto mount any attached devices. Helix also has a special Windows autorun side for Incident Response and Forensics.
Bastille: Security hardening script for Linux, Mac OS X, and HP-UX - The Bastille Hardening program "locks down" an operating system, proactively configuring the system for increased security and decreasing its susceptibility to compromise. Bastille can also assess a system's current state of hardening, granularly reporting on each of the security settings with which it works. Bastille currently supports the Red Hat (Fedora Core, Enterprise, and Numbered/Classic), SUSE, Debian, Gentoo, and Mandrake disibutions, along with HP-UX and Mac OS X. Bastille's focuses on letting the system's user/adminisator choose exactly how to harden the operating system. In its default hardening mode, it interactively asks the user questions, explains the topics of those questions, and builds a policy based on the user's answers. It then applies the policy to the system. In its assessment mode, it builds a report intended to teach the user about available security settings as well as inform the user as to which settings have been tightened.
Acunetix Web Vulnerability Scanner: Commercial Web Vulnerability Scanner - Acunetix WVS automatically checks your web applications for vulnerabilities such as SQL Injection, cross site scripting, and weak password sength on authentication pages. Acunetix WVS boasts a comfor GUI and an ability to create professional website security audit reports.
trueCrypt: Open-Source Disk Encryption Software for Windows and Linux - trueCrypt is an excellent open source disk encryption system. Users can encrypt entire filesystems, which are then on-the-fly encrypted/decrypted as needed without user intervention beyond entering their passphrase intially. A clever hidden volume feature allows you to hide a 2nd layer of particularly sensitive content with plausible deniability about whether it exists. Then if you are forced to give up your passphrase, you give them the first-level secret. Even with that, attackers cannot prove that a second level key even exists.
Watchfire AppScan: Commercial Web Vulnerability Scanner - AppScan provides security testing throughout the application development lifecycle, easing unit testing and security assurance early in the development phase. Appscan scans for many common vulnerabilities, such as cross site scripting, HTTP response splitting, parameter tampering, hidden field manipulation, backdoors/debug options, buffer overflows and more.
N-Stealth: Web server scanner - N-Stealth is a commercial web server security scanner. It is generally updated more frequently than free web scanners such as Whisker/libwhisker and Nikto, but do take their web site with a grain of salt. The claims of "30,000 vulnerabilities and exploits" and "Dozens of vulnerability checks are added every day" are highly questionable. Also note that essentially all general VA tools such as Nessus, ISS Internet Scanner, Retina, SAINT, and Sara include web scanning components. They may not all be as up-to-date or flexible though. N-Stealth is Windows only and no source code is provided.
MBSA: Microsoft Baseline Security Analyzer - Microsoft Baseline Security Analyzer (MBSA) is an easy-to-use tool designed for the IT professional that helps small and medium-sized businesses determine their security state in accordance with Microsoft security recommendations and offers specific remediation guidance. Built on the Windows Update Agent and Microsoft Update infrasucture, MBSA ensures consistency with other Microsoft management products including Microsoft Update (MU), Windows Server Update Services (WSUS), Systems Management Server (SMS) and Microsoft Operations Manager (MOM). Apparently MBSA on average scans over 3 million computers each week.

COMPUTER SECURITY TOOLBOX originally appeared on AskApache.com

Hacking VLAN switched networks

AskApache — Sun, 30 Mar 2008 23:49:53 +0000

AskApache.com

I wanted to post this for those rare hackers and network admins out there trying to find vlan info. Even though vlans are rarer these days..There isn't much vlan info on the net in terms of specifics and I had to learn all about it because I needed to log in to a switch that was on a different vlan. Impossible? No it is really easy after the research.. It was beautiful when I finally got to that terminal login.

Update: There is a great resource for vlan information now available at Capturing VLAN Packets from the incomparable, uncommonly great open-source gem WireShark

Discussion I had with the developers of ettercap.

The original post is still up on the Ettercap Development Form

VLAN Virtual Lan. VLAN spoof. Layer-2 switch. wvlan. Hello there, first post here. First I want to thank all of the developers for an outstanding product. Also I notice that you freely give your time to peruse through the forums and help people. Outstanding.. My problem is this. I am connected to an Allied Telesyn 8024 layer 2 switch (http://alliedtelesyn.com/products/details.aspx?9) with port security enabled to the Secure mode. By misconfiguration, only the switch itself is on a vlan. The router is not, the other switches in the network are also not. No ports are included in this vlan on the switch. The management vlan is 1. The switch checks its CAM table for the correct mac from the correct port before it forwards the frame. This means that a remote management session is highly improbable from the current setup (static mapping). The only response I can get from the switch is with an arp request. For this I use the arping utility for linux. The switch is on vlan 1. Here is an ethereal capture of an arp reply from the switch.

No. Time Source Destination Protocol Info
3 1.500954 AlliedTe_XX:XX:XX LinksysG_XX:XX:XX 0x0806 ARP
 
Frame 3 (60 bytes on wire, 60 bytes captured)
Arrival Time: May 29, 2004 15:21:41.982501000
Time delta from previous packet: 0.001966000 seconds
Time since reference or first frame: 1.500954000 seconds
Frame Number: 3
Packet Length: 60 bytes
Capture Length: 60 bytes
Ethernet II, Src: 00:30:84:XX:XX:XX, Dst: 00:04:5a:XX:XX:XX
Destination: 00:04:5a:XX:XX:XX (LinksysG_XX:XX:XX)
Source: 00:30:84:XX:XX:XX (AlliedTe_XX:XX:XX)
Type: 802.1Q Virtual LAN (0x8100)
802.1q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 0000 0000 0001 = ID: 1
Type: ARP (0x0806)
Data (42 bytes)
 
0000 00 04 5a XX XX XX 00 30 84 XX XX XX 81 00 00 01 ..Zj...0.M. ....
0010 08 06 00 01 08 00 06 04 00 02 00 30 84 XX XX XX ...........0.M.
0020 ac 10 00 05 00 04 5a XX XX XX ac 10 00 80 00 00 ......Zj........
0030 00 00 00 00 00 00 00 00 00 00 00 00 ............

The switch connects to the router through port 24. Enhanced Stacking is enabled and this switch is set up as a master. It can only see 1 other switch. The second switch is between the first switch and the router. I am unable to connect to the second switch or get any type of response. This is important because if I could connect to the second switch via telnet or web management, I could set it up as a master and reconfigure switch one. So keep that in mind.

I am able to use ettercap to arp poison all of the other hosts connected to the switch. I am not able to see the rest of the net. I know the rest of the net is there from broadcast arps from the router. I also have the macs and IPS of most of them.

Would it be possible to DoS the switch? Any recommendations? What about some type of vlan packet DoSer.

Could I use ifconfig to spoof my mac and ip address to look like an allied telesyn switch, and then generate packets with the correct vlan header to access the switch management ports?

How can I generate vlan packets (libnids, libnet, libdnet) and use them with ettercap?

What, as developers, are you thinking about vlans? As they are becoming increasingly used. Don't tell me this is the beginning of the end for ettercap!

Trunking is not enabled. Spanning tree is not enabled.

... with port security enabled to the Secure mode.

Looking at the docs, there are three modes of Port Security on your switch. In "Limited mode" the port is only going to allow a preset number of mac address to be learned from each port. Anything else is dropped. If that's the case, see if you can get into another port that may be misconfigured. This port could be used by a second NIC to send out the spoofed ARPs. This isn't built into ettercap but nemesis would do the trick. Just write a shell script to repeat the ARP every 10 seconds or so.

Another thing you may want to look into is becoming a trunk port. Chances are your admins have removed this function as well but be careful, apparently the switch can only handle one trunk at a time (what a cheap switch!).

When all else fails find the console, hook up a serial cable, reset it and hit enter while it's booting. type "boot" at the prompt and then put the password of "admin" in.

(WARNING - all advise here can get you into deep doodoo. If you're not authorized to do any of these things and you get caught, you're SOL)

I really think your best option is a second NIC with nemesis. If port security is enabled then you're really kinda stuck. Can't overflow, can't mitm.

Solution

Whoah...

I was going to try the various types of attacks in those articles, and had the libnet codes all ready, but then I decided to look here http://www.candelatech.com/~greear/vlan.html

I recompiled my kernel with 8021q as a loadable module. (included with my kernel)

I edited my rc.d/rc.local file with the lines

/sbin/i forget it loads the module dependencies.
 
/sbin/modprobe -a 8021q

and then rebooted, I installed the vconfig utility (rpmfind.net) and then did

$ vconfig add eth0 1 (1 is the vlan id I needed) This created device eth0.1
 
$ ifconfig eth0.1 netmask 255.255.255.0 broadcast 255.255.255.255 hw ether (mac from eth0 becasue port security is on and drops all other macs) (IP but different from eth0)
 
$ ifconfig eth0.1 up

Now, I tried to ping the switch and examined packets in ethereal on the "any" device.

$ ping switch  //didn't work.
 
$ ping -I eth0.1 switch //Bingo! It responded!
 
$ telnet -b eth0.1 switch

Bingo! I then configured the switch to operate correctly, saved changes, and quit. I rmmod 8120q and edited rc.d file. Fixed.

Standard Port security has nothing to do with arp poisoning, because "spoofed" arp packets have the right source mac address of the ettercap machine.

Well the switch had a predefined static mac address that was allowed on each individual port. IOW, I couldn't connect to anything with a mac other than the one predefined. Not even by spoofing other legal macs located on different ports.

Maybe you guys should look into a plugin that spoofs a vlan header so you could arp poison different vlans?

NaGA wrote: Standard Port security has nothing to do with arp poisoning, because "spoofed" arp packets have the right source mac address of the ettercap machine. D'oh, you're right. My brain wasn't engaged while replying. I was thinking about something entirely different.

ALoR wrote:
Maybe you guys should look into a plugin that spoofs a vlan header so you could arp poison different vlans?

you can use the in-kernel support for 802.1q and setup a virtual NIC on that vlan, then use the -i to select it as the default interface

Downloads and Documents

Ive googled the sauce out of vlan info and these are the best docs I've come across. It includes actual code used with libnet to attempt the various types of attacks with vlans. A must read.

Making unidirectional VLAN and PVLAN jumping bidirectional

Advisory: Making unidirectional VLAN and PVLAN jumping bidirectional Class: design bug Vulnerable protocols: 802.1q, various PVLAN implementations Model Specific: This is a protocol, and not vendor-specific attack DETAILS: Wepwedgie, a tool by Anton Rager for traffic injection on 802.11 networks protected by WEP, solves the problem of unidirectional communication by bouncing packets from the target host to a third external host under the attackers control. We employ exactly the same principle to bypass both VLAN and PVLAN network segmentation. 1. Modification of the double-tagging VLAN jumping attack. The attacker tags his malicious data with two 802.1q tags and sends the packet with a spoofed source IP of a host under his or her control. This can be any host to which a valid route from the target VLAN is present, including an external host on the Internet. The first tag gets stripped by the switch the attacker is plugged into and the packet is forwarded to the next switch. The remaining tag contains a different VLAN number, to which the packet is sent. So, data is forced to pass between the VLANs. The receiving host will check the source IP of the arriving packet and send the reply to this IP, which is a host that belongs to the attacker. This attack can be launched using Yersinia (http://sourceforge.net/projects/yersinia/). 2. Modification of the MAC spoofing PVLAN jumping attack. The attacker sends a packet with a valid source MAC but a spoofed source IP of a host under his or her control. This can be any host to which a valid route from the target PVLAN is present, including an external host on the Internet. The target MAC address is replaced with the one of a gateway router. A switch would forward such packet to the router, which will then look at the IP and direct the packet to the target. Of course, the source MAC of the packet will be replaced by the one of the router, which would then direct the reply packet from the target to the host that belongs to the attacker. This attack can be launched using pvlan.c from the Steve A. Rouiller's "Virtual LAN Security: weaknesses and countermeasures" GIAC Security Essentials Practical Assignment. Note: Such attacks can be used for different purposes from portscanning to communicating with a backdoor on a different VLAN or PVLAN. Risk Factor: Medium Workarounds: There are no direct workarounds. Implement strict egress filtering against the spoofed packets described.

Interview of ALoR and NaGA

Here is a quick interview with Alberto Ornaghi (a.k.a. ALoR) and Marco Valleri (a.k.a. NaGA). Each is 26 years old, and they work as security consultants for two different company in Milan. http://www.newsforge.com/article.pl?sid=04/11/04/2114203

VLAN Setup and Configuration for virtual VLAN device

802.1Q VLAN implementation for Linux

To get started, you will want to download the latest vlan.X.X.tar.gz file (to your $HOME directory.) Unpack it with your favorite commands, for example: tar -xvzf vlan.1.6.tar.gz Alternatively, you can get it from the CVS Repository using something like this:

Install and configure on your machine.
Specify the vlan repository: export CVSROOT=:pserver:anonymous@cvs.candelatech.com:/home/cvs_user/vlan
Log in to the repository: cvs login (PASSWORD: anonymous)
Check out the source: mkdir vlan; cd vlan; cvs -z3 checkout vlan

Now, you should have a vlan directory in your home directory. You only have to patch the kernel if you are using Linux 2.4.14 or earlier. Now, read the README or other docs to figure out what kernel it patches against. A list of mirrors are kept at www.kernel.org. Unzip and un-tar this in your home directory as well, which should create a linux directory in your $HOME directory. Example: tar -xvzf linux-2.2.14.tar.gz Now add the VLAN kernel changes to the kernel if your kernel requires it. I finally figured out how to do patches that diff can handle (I think I did it right at least!). You will find the patch in the vlan directory. It will be called: vlan.patch, or something equally straight-foward. Apply the patch to your kernel:

cd $HOME/linux
patch -p 1 < $HOME/vlan/[vlan.patch]

Your new, patched, kernel should be in your INCLUDE path before trying to compile the vconfig program. One way to get things working is to link $HOME/linux to the 'linux' directory that you just un-zipped and patched. A command might be something like: cd $HOME; ln -s /home/greear/kernel/2.4/linux.dev linux Build the vconfig program in the $HOME/vlan directory:

cd $HOME/vlan
make

Now, time to compile your new kernel! Use the make xconfig command in your $HOME/linux directory to select your kernel options. The option related to 802.1Q VLANs is found under the Networking options. If the option is not highlighted, make sure you select "Experimental Drivers" in one of the first xconfig menus. Assuming your kernel compiled cleanly (yell if it didn't and you think my code broke it!!), you are now ready to try it out!! Install your kernel in the normal manner (fix up your /etc/lilo.conf file appropriately and run lilo as root.) Reboot your computer and choose your new kernel. As your computer comes back to life, there will be little sign that you are now 802.1Q capable, other than a line spit out during the boot process. There should be a config programs in your $HOME/vlan directory: vconfig. vconfig is used to create and destroy VLAN devices. So, lets create a VLAN device on your first ethernet NIC. vconfig will list a short spiel on how to use it. The vconfig command I usually use is:

vconfig add eth0 5

This attempts to create a VLAN device with VLAN-ID of 5 on the eth0 device. If you want to delete a VLAN, use something like:

vconfig rem eth0.5

You will also need to give it an ip, eg: ifconfig -i eth0.5 192.168.2.1 and configure it UP: ifconfig -i eth0.5 up NOTE: You can get lots of VLAN related configuration information from the /proc/net/vlan/* files by using 'cat' or 'more' to look at them.

Intrusion Detection FAQ: Are there Vulnerabilites in VLAN Implementations? VLAN Security Test Report

David Taylor July 12, 2000

Introduction

This document details the testing methodology and results of security testing conducted against Cisco's VLAN implementation. The results presented were gathered entirely from this testing. The applicability of the results to other Cisco switch models, or other vendors' products is limited.

Background

Virtual LAN (VLAN) technology is used to create logically separate LANs on the same physical switch. Each port of the switch is assigned to a VLAN. In the case of the Cisco Catalyst, VLAN'ing is done at layer 2 of the OSI network model, which means that a layer 3 device (router) is required to get traffic between VLANs (possibly a filtering device). In addition to the above, VLANs may be extended beyond a single switch through the use of trunking between the switches. The trunk allows VLANs to exist on multiple switches. To preserve VLAN information across the trunk, the ethernet frame is 'wrapped' in a trunking protocol. Cisco have their own proprietary trunking protocol, but they also support the emerging 802.1q standard - we used 802.1q trunking in these tests.

Equipment

The following equipment and software was used during testing.

2 x Cisco 2924 Ethernet switches. Each with 24 x 10/100 UTP
2 x Generic PCs. Each with one 10 Mbit ethernet NIC.
1 x UTP crossover cable for trunking
2 x UTP cables for PC connectivity
Switch console cable and PC adaptors
Windows NT operating system on both PCs
Network Associates' Sniffer Pro v.2.0.01

Preparation

The switches were both prepared with a similar configuration. Ports 1-8 were assigned to VLAN 1. Ports 9-16 were assigned to VLAN 2. Ports 17-23 were assigned to VLAN 3. Port 24 was designated as an 802.1q trunk port. A sample switch configuration can be found in Appendix A. The two PCs were configured with IP addresses on the same C class subnet. This is largely irrelevant, due to the VLAN'ing being implemented at OSI layer 2, but it did remove the ICMP net unreachables and other errors that would otherwise have been experienced during the testing.

Methodology

Sample frame capture The two PCs were attached to the same VLAN of one of the switches. An ICMP echo (ping) packet was sent from PC 1 to PC 2. This was captured with Sniffer Pro on PC 2 and the packet was viewed in raw hex. This packet was recorded for further use in testing. The packet generation component of Sniffer Pro was started on PC 1 and the packet data captured above was entered into the tool. It is important to note that the packet generation tool in Sniffer Pro takes care of the ethernet preamble and frame check sequence. When the data was fully entered, the constructed packet was sent from PC 1 to PC 2, and was captured on PC 2 to ensure that it was correctly identified. Insertion of 802.1q tag Next, PC 2 was shifted to port 24 on switch 1 (the trunk port) and the sniffer software was started on this machine. PC 1 was left on a VLAN 1 port of the same switch. From the command prompt on PC 1, a non-existent IP address was pinged. As this non-existent IP address did not have an entry in PC 1 ARP table, the machine broadcast an ARP lookup and this lookup was captured on PC 2. As PC 2 was listening on a trunk port, it received the ARP lookup in 802.1q format, containing the 4 byte 802.1q tag. This process was repeated, with PC 1 moved to a VLAN 2 port and from these two captures, the format of the 802.1q tag was found to be "81 00 0n nn", where nnn is the VLAN number. For example, frames on VLAN 1 would have a tag of "81 00 00 01", frames on VLAN 2 would have a tag of "81 00 00 02". The 802.1q tag is positioned directly after the source MAC address of the frame and before any of the IP header information. Taking into account this frame format and placement information, an 802.1q tag was inserted into the ethernet frame that was generated in the previous section. 802.1q Frames into non-trunk ports For the next test, PC 2 was moved to a VLAN 1 port on the second switch. PC 1 was moved to a VLAN 1 port on the first switch. The trunk cable between the two switches was reconnected. The 802.1q frame generated in Sniffer Pro was sent from PC 1 and was received by PC 2 as a plain ICMP echo ethernet frame, without the 802.1q tag. This test was repeated with both PCs on VLANs 2 and 3 also. In each case, the handcrafted frame was delivered to the destination machine. Hopping VLANs For the next test, the PCs were connected to different VLANs on each of the switches and an attempt was made to get the generated frame to ˜hop' from one VLAN to the other. Various VLAN ID's were used in an effort to cover as many combinations as possible. Additionally, attempts were made to get frames to hop VLAN boundaries within the same physical switch. The following results were collected. Different Switches The two combinations of note are in the first table (different switches) where traffic was sent from VLAN 1 to VLAN 2, and from VLAN 1 to VLAN 3. Native VLAN of trunk port Following the previous test, prolonged discussions took place with the switch vendor to discuss the implications of the results above. After consultation with their developers it was concluded that the traffic from VLAN 1 was allowed to hop to other VLANs because the trunk port was also set (implicitly) to native VLAN 1. They suggested that by changing the native VLAN of the trunk port the VLAN hopping could be eliminated. This was tested and was found to be true. The results are shown below. In each case, the tag VLAN ID was set to the destination VLAN (This was found to be the most likely to succeed in preceding tests).

Implications

In a default configuration it is possible to inject 802.1q frames into non-trunk ports on a switch and have these frames delivered to the destination.
It is possible to get 802.1q frames to hop from one VLAN to another if the frames are injected into a switch port belonging to the native VLAN of the trunk port. It is also necessary for the source and destination ethernet devices to be on different switches.

This vulnerability could be exploited if the following conditions were met:

The attacker has access to a switch port on the same VLAN as the native VLAN of the trunk port.
The target machine is on a different switch in the same trunk group.
The attacker knows the MAC address of the target machine.
Some layer 3 device exists to provide a connection from the target VLAN back to the source VLAN.

Unconfirmed Findings

In our discussions with Cisco they stated that this issue was present in all of their VLAN switches and all of the competitors switches that they tried. This is assumed to include Nortel and 3Com devices.

Recommendations

Try not to use VLANs as a mechanism for enforcing security policy. They are great for segmenting networks, reducing broadcasts and collisions and so forth, but not as a security tool. If you MUST use them in a security context, ensure that the trunking ports have a unique native VLAN number.

Hacking VLAN switched networks originally appeared on AskApache.com

Pen-testing Security Tools

AskApache — Fri, 02 Nov 2007 04:37:31 +0000

AskApache.com

While testing the exploitability of your target and mapping out vulnerabilities it is important to gain access inside the targets defenses so that you can establish an internal foothold like a owned box or switch. This is so you can use a tool to discover the packet-filtering being used, and literally map out the firewall/IDS rules. Needless to say that really provides you with a lot more complete vulnerability assessment to help discover more weak spots in the system.

Socat
Socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (terminal or modem, etc.), socket (Unix, IP4, IP6 - raw, UDP, TCP), SSL, a client for SOCKS4, or proxy CONNECT. It supports broadcasts and multicasts, abstract Unix sockets, Linux tun/tap, GNU readline, and PTYs. It provides forking, logging, and dumping and different modes for interprocess communication. Many options are available for tuning socat and its channels. Socat can be used, for example, as a TCP relay (one-shot or daemon), as a daemon-based socksifier, as a shell interface to Unix sockets, as an IP6 relay, or for redirecting TCP-oriented programs to a serial line.
Samplicator
UDP Samplicator receives UDP datagrams on a given port and resends those datagrams to a specified set of receivers. In addition, a sampling divisor N may be specified individually for each receiver, which will then only receive one in N of the received packets.

ftester - for master pentesters only--get the lowdown on your packetfiltering rwhois - really great addition to using whois. Get additional info not on whois, query rwhois servers. lft - useful alternative method of tracerouteing. oppleman packit - define (spoof) nearly all TCP, UDP, ICMP, IP, ARP, RARP, and Ethernet header options etherape - really cool graphical program that displays connections and protocols similar to cheops. amap - fingerprinting xprobe2 - fingerprinting p0f2 - really exceptional fingerprinting. can be passively run in the BG. firewalk - good packetfiltering enumerator BGPview - bgp anyone? icmpenum - icmp fingerprinting dnstracer - awesome and creative graphical output of dns ssldump - not really that useful but impressive in a report mtr - alternative traceroute MRTG - favorite tool of ISPs, many uses here host - don't forget this one ike-scan - scan for vpns upnpscan - scan for upnp devices ftp-spider - get info on ftp server traceproto - very nice alternative to traceroute/firewalk

Also see: Vulnerability Scanner Review

sing
netleak
dmitry
isic
dnsa
nemesis
sara
zodiacdns
fragroute
sentry 2.0
Caecus
C-Parse
pchar
nmbscan
nbtscan
admsmb

Pen-testing Security Tools originally appeared on AskApache.com

Top 5 Vulnerability Port Scanners

AskApache — Mon, 22 Jan 2007 19:59:13 +0000

AskApache.com

Asked some hackers and Computer Security gurus to list their 5 favorite Vulnerability/Port Scanners. Here are the results.

You might like the more in-depth article: Vulnerability Scanners Review, or you may be looking for the Computer Security Toolbox

netcat
nmap
hping3
isic
p0f2
sing
nikto
firewalk - Detecting Firewall Rulles
nikto/wnikto - cgi scanner
whisker - cgi scanner with IDS feature
nessus / NewT - Vulnerability scanner
Qualys
eEye Retina
GFI Languard
ISS
Shadow Security Scanner (SSS)
pakketo's scanrand
FX Scanner
Acunetix web vulnerability scanner
IPCScan
SQLRecon
Metasploit
Whax
SoftPerfect Network Scanner - Network Share Scanner
Cain & Abel
autoscan - . integrate some other tools
hydra - Password finder by Brute forcer
Retina - Vulnerability scanner
X-scan3 - Vulnerability scanner GUI & CLI uses nasl
SupperScan - fast port scanner
Dfind - CLI vuln scanner
Sfind - CLI vuln scanner
scan1000 - fast CLI vuln scanner
ScanLine - CLI vuln scanner
GhostPortScanner (GPS) to keep from triggering firewalls
Superscan 3.0 for TCP Scanning
Superscan 4.0 for UDP Scanning
X-Scan v2.3 (nmap + nessus for windows)
Webdavscan.exe for Mass Port 80 Banner
NTscan for IPC$ Passwords
foundstone tools
sara
X-Scan
FOUNDSCAN
HSCAN
cheops-ng
dsns
MicrosoftBaseLine - IIS vuln scanner

Top 5 Vulnerability Port Scanners originally appeared on AskApache.com

Vulnerability Scanners Review

AskApache — Sun, 31 Dec 2006 04:39:00 +0000

AskApache.com

A few months back I did some intense testing of all the best vulnerability scanners out there.. I had a couple unix boxes hooked up, as well as some windows machines, and figured I could add clients to a "once-a-week" scanning contract. So naturally, I wanted to use the scanner that was the best for my purpose.

You might be looking for the article: Top 5 best Vulnerability Port scanners

I tested the following (trying to only list automated vulnerability scanners):

ISS Internet Security Systems
SSS Shadow Security Scanner
Retina eEye
Nessus
GFI Languard Network Security Scanner
Qualys www.qualys.com
Nstealth Security Scanner www.nstalker.com
Nikto
Whisker
Infiltrator infiltration-systems.com
Nscan

I was looking at 3 main areas while evaluating the scanners.

Comprehensiveness of the testing: including how many options are allowed for different scanning, IDS evasion, and types of scans. Also in this category is the availability for the latest exploits and a custom exploit option to allow me to plug in custom exploits.
Quality of the program: included in this category is availability of updates, speed of various variables, efficiency, "smartness" or "AI" of the program while scanning/reporting, security- (does running this version of this vuln scanner leave me vulnerable?), scheduling capabilities, alert and message capabilities, quaility of exploits, reactions to "false positives", and overall feature and capabilities.
Reporting Capabilities: How easy is it to create a report? The quality and design of the report. The comprehensiveness and personalization of the reports..

My findings

All of these programs can be tested for free, either through an evaluation or trial. I believe that the availability of these programs on the net (cracked versions) represents the conspiracy to aid script-kiddies everywhere so that these companies will then profit after an intrusion (or even a loud scan). Any of these programs are not to be used for cracking... Nothing serious at least.. using these programs is akin to knocking on every window of a house, while simultaneously ringing the doorbell and playing loud music in the driveway.. Some of the programs allow for easy use of anonymous socks and web proxys, but we all know by now that to achieve any real anonymity takes much more work than that.

After several months testing

It became clear to me which ones I liked the best. The top pick all-around (for windows) has to be retina's eeye, the company has a bunch of hackers and you can tell by their awesome product. Another great scanner (windows -web) are the different products offered by nstalker... very good web vuln scanner. Free for download.. A close 2nd would be the SSS, I love it, its great, but eEye's is better. The best choice for me however is nessus. Although it has a few negatives compared to some of the other products, it is the best solution for anything serious. Set-up up a couple of nessus servers around the globe and try attacking the same location through different networks for some real eye-opening fun. Also good to practice tracking and tracing. Qualys is poised to be super-rich, just check out their site... great if you don't want to mess with a scanner yourself. ISS is way to expensive and I didn't like the fact that I had to try for awhile until I could download it. But it is considered to be the best scanner and I admit the anti-pirating features and licensing of their software is pretty darn good. GFI is another top contender. It was just prone to crashing... A money place to get some other cool tools (Especially a professional SSL encryption tester) are available for free at www.foundstone.com. Get them soon, make a foundstone toolkit.... mcafee bought them so expect the free stuff to get yanked shortly... (I can't believe foundstone is almost gone...) Unless you are using these tools on yourself, or to check your friends pc or company (use scheduling with alerts to email or pager), there is no sane or logical reason to use these tools on any pc that you do not have express permission to use against.

Remember these tools are the noisiest tools in a security testers arsenal and if you use it just once on an unknown host, your ISP can yank your account, you can face legal action, etc.. Better to use firewalk, hping3 (now with scripting!), nmap, etc, and leave these crutch-like tools alone. Also check out www.coresecurity.com , for the best product I have found so far.

A few pointers..

If you have a network of your own, and you don't have the time or resources to set up a nessus installation at a remote site to repeatedly test your network, the best solution IME is Qualys. I am not affiliated in any way, I checked it out merely to examine the competition in the marketplace... after experiencing their free trial, unheard of amounts of documentation, and extremely skilled marketing team, I am still very jealous. What an easy concept but they took it to that level and it stands alone, IMHO. So if you need that kind of simple, safe, comprehensive fix, check out qualys. Still, you would ultimately save a lot of money (even after computing time and costs) by setting up your own remote server somewhere to automatically check the network for you. Multiple locations of course are ideal (go through different networks etc..) The major lack of enthusiasm I have for most of these "vulnerability scanners" is the fact that so many of them run on windows and nothing else. The benefits to running from (almost anything else) a linux type platform are enormous. IOW, nessus can only continue to get better and better, while the only advantage eeye, ISS, and the rest of the dozer boys have is better knowledge of exploits. This is really a small point (in the big picture) because of the fact that it is sooo easy to copy an exploit (for those that do that sort of thing). So if you are out there, go help out the excellent people who give us nessus, go give them a donation, help them develop the software, test it out and email them detailed feedback of your likes and dislikes, do what you can to keep it around. I see it as having the most potential. There are also vuln scanners that operate locally, many professionals use these to scan a lan of windows hosts for example. These are hard to find, so if you know of any, post.

Also, after months of testing all the different vuln tools, including tons not listed above, still failed to find a bug that allowed me root on a particular server I was testing for a friend. It was the mole.cfm script that finally got me in... an old exploit that none of the scanners tried. So if you are serious about keeping your data safe, hire a real expert (like foundstone or coresecurity) and do not rely on a commericial marketing solution alone.

Other Good Vulnerability Scanner Review here.. (12+ pages)

Vulnerability Scanners Review originally appeared on AskApache.com

AskApache » Search Results » vulnerability

30x Faster Cache and Site Speed with TMPFS

Introducing tmpfs

Beware of WebHosts

Hosting Company Tricks

Why it gets Evil

Where it gets Evil

Where it gets Fun (for me)

TMPFS kills the RAMDISK

TMPFS uses RAM+SWAP

Swap

Using TMPFS for Cache

Boosting Cache with TMPFS

Create TMPFS at wp-content/cache

Restoring TMPFS across Reboots

Mirroring TMPFS to Disk

/tmp, /var/run, and /var/lock

Resize /dev/shm

Securing and Using /tmp

tmpfs mount parameters

Using tmpfs for /tmp storage

Bind Mounts

Bind mounting and /dev/shm

/etc/default/tmpfs WorkAround

RSYNC vs. CP

Mount Options

Filesystems

/etc/fstab

/etc/fstab

The first field, (fs_spec)

The second field, (fs_file)

The fourth field, (fs_mntops)

The fifth field, (fs_freq)

The sixth field, (fs_passno)

More Reading

Use tmpfs for MySQL

Shell Script for Firefox tmpfs

COMPUTER SECURITY TOOLBOX

Obscure/Rare Security Software

More common security programs

Hacking VLAN switched networks

Discussion I had with the developers of ettercap.

Solution

Downloads and Documents

Making unidirectional VLAN and PVLAN jumping bidirectional

Interview of ALoR and NaGA

Other Links

VLAN Setup and Configuration for virtual VLAN device

802.1Q VLAN implementation for Linux

Intrusion Detection FAQ: Are there Vulnerabilites in VLAN Implementations? VLAN Security Test Report

Introduction

Background

Equipment

Preparation

Methodology

Implications

Unconfirmed Findings

Recommendations

Pen-testing Security Tools

Top 5 Vulnerability Port Scanners

Vulnerability Scanners Review

My findings

After several months testing

A few pointers..