Ok I just came back up to write the intro.. I'm trying to keep it short to avoid getting bogged down by the coolness of each step. Here is what goes on. When I logon to my XP machine at work, I bring my usb key and plug it in first. On logging a window pops up first and it's a password prompt to mount my encrypted drive leonardo. It also checks a keyfile that is located on my usb key, but all I do now is type in my password. That causes my encrypted folder to be accessible to me like a normal drive, and it autoruns a startup batch file. The batch file causes Portable versions of Firefox (all my bookmarks, my settings) to load, and launches Portable Mozilla Thunderbird (IMAP makes this work well), which is my favorite program (great GPG features and open-source!). Also Some Adobe CS4 software is loaded from the hard drive, like DreamWeaver.

The batch file also runs PortaPuttY plink to create forwarded tunnels from various remote servers and accounts, all using key-based encryption. This includes dynamic SOCKS 4/5 tunnels, VPN tun device tunnels, and of course the basic SSH port forwarding tunnels that are so powerful. These tunnels are automatically reconnected if they are disconnected, using simple windows builtin command-line tools. And believe me it was not easy to figure out how to make this all work using plink ( essentially the same as putty minus the gui ), I literally had to use almost all of my Windows kung fu to finally end up with this.

Using MyEnTunnel

Initially I was using the MyEnTunnel program combined with a custom windows batch install script I wrote to handle the tunnels.

The tunnels are very important to making things easy while improving security. It's not easy to understand at first, but basically it means you can now connect to ANY IP address:port as if you were on that very machine connecting to localhost, like if you pinged yourself!. The result is any traffic you want is now encrypted, and you can set up your servers to only accept connections from localhost, which could save you tons of memory, bandwidth, and security attack vectors to think about. So I configure everything to use these tunnels as proxies, like Mozilla Thunderbird and Chrome, Firefox, Pidgeon, all portable versions and running from my encrypted usb drive.

This means you can walk into my house with that usb key, plug into any computer here, and surf the web/check your emails all across SSH... I know for a fact I wouldn't be able to snoop that traffic! There is a lot of exciting things going on around here, new servers and all.. Its going to take a couple more posts for me to finish this up, enjoy the article and comment.

Buy a couple USB Mini Drives

The first thing to do, is purchase a USB thumb drive.. My favorite store, TigerDirect.com, has over 104 tiny usb drives for under $24.. I've used them since the late 90's.

I bought some 4GB PNY's the size of a fingernail at a gas station and they are amazing, way faster than say a dvd drive. Just try to do some research of the differences between the 16GB vs the $4 1GB drives.. You want speed because the whole drive will be encrypted. If you can afford the super excellent and crazy fast ones, hey send me one! Buying cheap means you can buy 3 or 4 so you can always have backups. This device will make you Internationally mobile, untethered from a box, maybe getting some work done at a cafe in Florenze, or at a beach hotel in Miami. Keep dreaming, but that is more possible with a better organized system.

Backup the USB Drive

You only need to know 1 way that works, there are several. The way I do backups is to copy the entire disk image of the usb, that way I can always access it in case of usb key failure, which does happen. Free software like CloneZilla live CD with its crazy cluster computing power, or Self Image, which is free for both linux and Windows. And you could never go wrong with GHOST, one of the first to make mega bucks in the market.. it's some seriously impressive software but not open-source. Even easier for some is to just set a cron job for dd to pipe the entire drive image to a remote computer using netcat, or sshfs, or curlftpfs, or just simple ssh like below. Once setup (without stupid, bulky, dangerous software), the files on your encrypted usb don't change often, otherwise I would want to sync a backup to happen automatically every X number of logins or days (test logfile time in bash_profile?)..

SSH Back-ups To Remote Server

Files and data on your drives slow it down tremendously, meaning a web server storing backups locally is slower than one storing them externally.

Notice how much safer this command is by optimizing both the CPU and DISK I/O.. Though it's much smarter to create a new separate ssh user, one with no shell and a passwordless safer key-based encryption. Then in your /etc/security/limits.conf file or your initscript you can cause that user to have nice -19 and ionice -c2 -n7 priority set all the time automatically, since sshd, compression, and disk writing are this accounts only job. turboslow is an alias defined in a ssh_config file so you don't have to type the host, port, and settings each time.

#
# much better ways to do this on google!!!!!!!
#
ionice -c2 -n7 nice -n 19 dd if=/dev/sdb2 bs=1k conv=sync,noerror | gzip -c | ssh turboslow "dd of=sdb2.gz bs=1k"

Note that you may decide it would be better to configure the ssh connection to a less CPU intensive algorithm, perhaps even protocol 1 and DES. That's perfectly alright, but the tradeoff is that the encryption can be broken much quicker, and so you would have to implement a cron job to create new keys on both ends of the tunnel every few hours.. It's really not a big deal to setup, kind of sweet way to use key-based encryption. Also, important files ( those containing passwords, any database ) are encrypted before transport using private GPG keys, which don't need to be changed. The other thing to think about too is only letting your main PC send/write on the backup host, so the backup host is only authorized to rx and can never login back to yours.

Hey! the Internet is a dangerous place you better believe it! And it's only going to get more interesting with cloud computing's breakthrough's... More people who know they're way around... I can always use an extra server, I'd love to expand my network another node without having to pay for it (free cloud computing?), so make sure your servers are locked up strenuously. Not super perfect, just a little unique or creative in your defense to avoid any coming super-worm's that may be employing vast arsenals of the deadliest attack-engines like metasploit.. Scarry rumors.

Compression Speeds: PBZip2, Rzip, Lzop, Gzip

Probably the fastest is to use rsync over ssh, which is what I'm doing, since the algorithms used by rsync are much faster and safer. Rsync also lets you specify a compression program, so depending on your machine you will want pbzip2 (for multi processors) or rzip which are the 2 fastest I know of, though I have had some reliability issues with rzip for gigabyte transfers. Pbzip2 is amazing, blew me away the first time being 8x faster (8 CPUs) then anything. You can get it and compile a static binary for your thumb drive if want at Parallel BZIP2 (PBZIP2). Heavy code, re: this note by Jeff Gilchrist

tar cpf "$G" --use-compress-prog=pbzip2 ./

Benchmarking for Performance

Finally a couple tips, you should get an idea what the device can do, format it a few times for linux and test it on windows, and vice versa.. Some drives are too small or too old and can only support fat32 filesystems on winblows, you DO NOT want fat32 because this drive is going to be 100% encrypted and then 100% transparently decrypted as you use it,

# note this is 512MB
dd if=/dev/sda1 of=/dev/null bs=512 count=1000000
512000000 bytes (512 MB) copied, 5.16588 s, 99.1 MB/s

Part II: Encrypted AutoRunning USB Key with TrueCrypt

Now this section anyone can do, it's so easy on Windows. What I'm going to show you how to do is get setup the right way super-fast. There are many ways to use TrueCrypt, it's one of the nicest built software programs's I've ever used... Sadly, it is not licensed open-source, and that is often a deal-breaker for security-conscious folks or anti-pirate anarchists. From the very helpful TrueCrypt web site:

• Creates a virtual encrypted disk within a file and mounts it as a real disk.
• Encrypts an entire partition or storage device such as USB flash drive or hard drive.
• Encrypts a partition or drive where Windows is installed (pre-boot authentication).
• Encryption is automatic, real-time (on-the-fly) and transparent.
• Parallelization and pipelining allow data to be read and written as fast as if the drive was not encrypted.
• Provides plausible deniability, in case an adversary forces you to reveal the password: Hidden volume (steganography) and hidden operating system.
• Encryption algorithms: AES-256, Serpent, and Twofish. Mode of operation: XTS.

Further Reading

• Network File Copy using SSH
• Check out the trunk version of PuTTY:~ svn co svn://svn.tartarus.org/sgt/putty

The real fun doesn't start till all the automation starts, automating all of that from a couple batch files I wrote, one click setup. Kind of like building your own knoppix for when you have to use Windows. To begin this tutorial, setup a truecrypt traveller setup on your usb and also install the portaputty package onto the usb. You do this by creating a 3GB or whatever file on the usb and then mounting that file like you would mount an iso file. I will show the Windows Batch file I use and the tricks with Windows Volume names and how to consistently make it all work. Then we will setup MyEnTunnel with a customized batch file that forces all puttys to use portaputty (sweet hack stolen from sysinternals pagedefrag tool).Stay Tuned!

PortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG originally appeared on AskApache.com

This is part 2 of my Windows XP Optimization article: Make Windows XP Blazingly Fast.

The first article was meant as a detailed and thorough introduction to speeding up Windows-based PC's in a way that makes it easy to follow, without getting too specific. So make sure you read that first, and pay the most attention to freeing up RAM, CPU, and Disk IO speed by reducing the number of services and processes that are running, we will deal with defragmentation, hard-drive speed, Disk IO, Prefetching, and Pagefile/Registry Defragmentation now.

This article has some really really really great stuff for ya. It shows which tools (all but one completely free) are the best and you will use them for a long time, they are all very good. That is just a side benefit, as this article is really more of a step-by-step guide to optimizing your system that won't have to be repeated for at least a year. The result of course is a much more responsive PC.

Whats New

After writing that article I continued my research and testing into the subject on my personal computers. I wanted to test out several additional programs and methods before I wrote about them for you guys, and I found a few really sweet additions that had a very big performance gain for all my computers, from my oldest and slowest PC's to my new 4K power laptop. This article is primarily focused on optimizing your hard drive data and improving your Disk IO speed, and you will definately see an improvement in speed. It doesn't get REALLY good until the defragmenting section..

1. Clean Up Hard Drive - Removing unneccessary files
2. Clean Registry - Fixing slow registry problems automatically
3. Ultimate Defragmenting - The best defrag method I use
4. Optimize Physical Hard Disk - Final step that cleans and heals your physical disk

Clean Up Hard Drive

The first step is to clean up all the extra, temporary, and unneccessary files cluttering your hard-drive. The reason is because we will be defragmenting your hard-drive like its never been defragged before, then we are going to go over every single bit and byte of your hard-drive to optimize the physical sectors and storage of your data.. Also we will be running a check of your registry and cleaning out bad links and other slow errors, so get it as clutter-free as possible. DON'T use Windows built-in folder compression, it makes defragmentation worse... DO use 7-zip or winrar to create a solid archive file of any misc directories with a bunch of files... You only need one program to clean your system.

CCleaner

CCleaner is a Small, Fast and Free software that removes unused and temporary files from your system and allows Windows to run faster, more efficiently and gives you more hard disk space. I've now been using it for several months and love it. As well as cleaning up old files and settings left by standard Windows components, CCleaner also cleans temporary files and recent file lists for many applications. Including: Firefox, Opera, Safari, Media Player, eMule, Kazaa, Google Toolbar, Netscape, Microsoft Office, Nero, Adobe Acrobat Reader, WinRAR, WinAce, WinZip and more... Google Chrome, Opera, Safari, etc..

• Recycle Bin, Clipboard
• Windows Temporary files, Windows Log files, Chkdsk file fragments
• Recent Documents (on the Start Menu), Run history (on the Start Menu)
• Windows XP Search Assistant history, old Prefetch data, Windows memory dumps after crashes

Download CCleaner

Registry Cleaning

The registry is Windows biggest mistake, (although I'm sure they like it), and basically holds all the information for your programs and Windows. Things like the size of your windows, recent file lists, icon files for different icons, etc.. I've never seen a computer that didn't have some registry issues, so this needs to be cleaned and may have a huge impact on your speed. Real quickly, here are some programs to backup and restore your registry, optimize and defrag your registry, and finally search and clean any errors in your registry.

CCleaner

Yup! CCleaner also takes care of most of the performance issues of your registry. It's very safe and fast. CCleaner uses an advanced Registry Cleaner to check for problems and inconsistencies. It checks the following:

• ClassIDs, ProgIDs, Application Paths, Icons
• Uninstallers, Shared DLLs, Fonts, Help File references
• File Extensions, ActiveX Controls, Invalid Shortcuts and more...

Download CCleaner

ERUNT - The Emergency Recovery Utility NT

ERUNT is a Registry Backup and Restore utility for Windows NT/2000/2003/XP. I use this to backup my registry automatically or on command.. Ive used it for years and it's always good to backup before you do anything.

# Here's the command I use (only for advanced users familiar with autoback)
"%ProgramFiles%\ERUNT\AUTOBACK.EXE" %SystemRoot%\ERDNT\#Date# sysreg curuser otherusers /noconfirmdelete /noprogresswindow /days:45 /alwayscreate

Download ERUNT · (Details)

NTGREGOPT - NT Registry Optimizer

NTGREGOPT is a Registry Optimization tool for Windows NT/2000/2003/XP/Vista that minimizes the size of your registry files by simply compacting the registry hives to the minimum size possible.

Registry files in an NT-based system can become fragmented over time, occupying more space on your hard disk than necessary and decreasing overall performance. You should use the NTREGOPT utility regularly, but especially after installing or uninstalling a program, to minimize the size of the registry files and optimize registry access. The program works by recreating each registry hive "from scratch", thus removing any slack space that may be left from previously modified or deleted keys. It does NOT change the contents of the registry in any way, nor does it physically defrag the registry files on the drive. I recommend using this once every couple weeks. I scheduled it to run automatically.

Download NTGREGOPT · (Details)

Ultimate Defragmenting

I say "Ultimate Defragmenting" because this is the result of a lot of testing of all the various defragmenting software out there, reading a lot of documentation, and running benchmarking to find the fastest results. This is a mix of several individual defragmenting steps combined for a once-a-year ultimate defragmenting session. This is what I use today, and although it's altogether a long process, each step you'll add a new tool or skill that you can use by itself from here on out.

ATTENTION: While running MyDefrag/JkDefrag, SpinRite, and UltraDefrag your computer can get very hot and that is very not cool. I set my laptop on a coke can and pointed a small desk fan at it which kept it very very cool, so do what you can to minimize heat during these programs.

PageDefrag

PageDefrag uses advanced techniques to provide you what commercial defragmenters cannot: the ability for you to see how fragmented your paging files and Registry hives are, and to defragment them. In addition, it defragments event log files and Windows 2000/XP hibernation files (where system memory is saved when you hibernate a laptop). One of the limitations of the Windows NT/2000 defragmentation interface is that it is not possible to defragment files that are open for exclusive access. Thus, standard defragmentation programs can neither show you how fragmented your paging files or Registry hives are, nor defragment them. Paging and Registry file fragmentation can be one of the leading causes of performance degradation related to file fragmentation in a system.

I personally keep this enabled for every boot, as it only takes a few seconds after the first time it's run.

Download PageDefrag

MyDefrag

JkDefrag is a disk defragmenter and optimizer for Windows 2000/2003/XP/Vista/2008/X64. Completely automatic and very easy to use, fast, low overhead, with several optimization strategies, and can handle floppies, USB disks, memory sticks, and anything else that looks like a disk to Windows. Included are a Windows version, a commandline version (for scheduling by the task scheduler or for use from administrator scripts), a screensaver version, a DLL library (for use from programming languages), versions for Windows X64, and the complete sources. (Frequently Asked Questions)

After trying out dozens of degragmenting programs, this is my favorite. I utilize the cool screensaver function and just run sometimes when I'm calling it a day.

Tips and tricks

• Many users start looking for defragmentation/optimization programs when their computer becomes slow. The main reason for a slow computer is a full harddisk. A full harddisk is slow because the distance between files is greater than on a fresh practically empty harddisk. Deleting half the data on a full disk will just about double the speed. The more free diskspace, the faster your computer will be.
• Buy a second harddisk (for example an USB harddisk) and move little used stuff from your primary harddisk to that secondary harddisk. The second disk can also be used for backing up the primary disk.
• When buying a new computer, buy the biggest harddisk you can afford. Investing in a bigger harddisk gives more speed-per-dollar than investing in a faster CPU or investing in more memory.
• Cleanup old junk from your harddisk before running MyDefrag. You can clean Windows files with for example "Start -> Programs -> Accessories -> System Tools -> Disk Cleanup", or with something like the freeware CCleaner program.
• Reboot before running MyDefrag. This will release files that are in use, so they can be defragmented and optimized.
• Boot into Windows safe mode by pressing F8 when booting, and then run MyDefrag. It will be slower because the Windows disk cache is off in safe mode, but MyDefrag will be able to process (a few) more files.
• Stop your real time virus scanner before running MyDefrag. Virus scanners check all disk activity, making defragmentation and optimization very slow.
• Move the swap file to another volume, reboot, defragment, and move the swap file back. If you don't have a second volume then temporarily make the swap file small, for example 100Mb.
• Package unused files with a packager such as 7-zip. The packagefile not only takes less harddisk space, but will also defragment and optimize much faster than the individual files. Note: This does not apply to Windows NTFS compression, which will actually make defragmentation and optimization slower.
• The first partition on a harddisk is significantly faster than other partitions. Try to use other partitions only for data that is used less often, such as music, movies, archives, backups, logfiles.
• If you have 2 physical harddisks of the same speed, then place the pagefile on the first partition of the second harddisk.

The way I recommend is to run MyDefrag at the highest level of defragmentation once, which took my fastest PC almost 30 hours. Once that is done you can just run it normally in 20 minutes or so.. This software also has the best defrag information I've found to date, so check out the documentation on the site.

Download MyDefrag

UltraDefrag

UltraDefrag is a powerful Open Source Defragmentation tool for Windows Platform. It is very fast, because the defragmenting is done via the kernel-mode driver. There are three interface available : Graphical, Console and Native. I personally like the MyDefrag more because I think it does a better job, but I also use the UltraDefrag tool because it has one very important feature like the PageDefrag tool. It has a native version. That means it can run before Windows loads up by utilizing the bootexecute, the same place that windows chkdsk runs at boot. It also can takeover for Windows builtin prefetcher, to speed up the loading of frequently used programs, which I'll explain a bit later.

I set ultradefrag up after the MyDefrag 30hour defrag completes, to run at boot and control the prefetching, then I erase any prefetch files currently saved and reboot which lets it defrag the system.

erase /Q "%SYSTEMROOT%\Prefetch\*.*"

Once both MyDefrag and UltraDefrag have run THEN I finally login to windows and don't open any programs to let the windows OS files get optimized by not doing anything at all for 5 minutes.. Then I reboot and log back in and this time I don't do anything for 30 minutes and reboot. Finally I log back in and this time I instantly load up 10 of my most frequently used programs, (DreamWeaver, Photoshop, Firefox, Chrome, Notepad2, Thunderbird, Internet Explorer, and a few others) and once they are all loaded I don't do anything for an hour. Then I reboot and repeat that same process.

This may seem odd or made up but I do my research and this allows your prefetched files to be optimized, including your boot prefetch files. Once that is done I reboot and run all the defrags again. Then I reboot and am ready for the last step.

Download UltraDefrag

Optimize Physical Hard Disk

Now at this point the system is defragged and optimized as much as I can get it, but the last step is to run a program to go over every single bit on our hard-drive-disk to keep the drive clean and healthy and its too technical for me to understand, I just know its amazingly cool and I noticed a big change right away.

HD Tune

HD Tune is a fantastic little utility that you can use to benchmark the DISK IO speed of your various drives, internal and external, fixed and USB, firewire, etc.. Other than using it to determine your fastest drives for moving your program files and temps to, I am just including it in this article because it is an awesome program that you will love.

Download HD Tune

SpinRite

This is the last step in this guide, and was the one thing that surprised me the most in terms of how much of a speed improvement I noticed after using it. SpinRite is the most capable, thorough, and reliable utility that has ever been created for the long term maintenance, recovery, and repair of mass storage systems. SpinRite is not a drive defragmenter. SpinRite operates with the drive's built-in intelligence to reassign and relocate defective sectors without creating file system fragments. Thus, running SpinRite does not create fragments, but neither does it eliminate any that may exist before it was run. Unlike any other disk utility, SpinRite interfaces directly to the hard disk system’s hardware, rather than working through the system’s operating system or BIOS. FAQ.

The way that we use SpinRite in this article is method #4, Drive Maintenance mode, which reads and writes and verifies every single sector and area of your hard drives, and improves the health of your hard drive a lot. Even on my new 4K dell power laptop, this had a noticeable improvement on speed. After running this 20+ hours for my fastest PC, I rebooted, defragged with jkdefrag, and that is the end of this article.

In case you don't already know . . . What is SpinRite?

SpinRite is a stand-alone DOS application that specializes in the recovery of marginally or completely unreadable hard and floppy disk data, and in the lifetime maintenance of PC mass storage devices. It earned its stripes many years ago by introducing the concept of non-destructive low-level reformatting and sector interleave optimization. Since then its capabilities have continued to broaden until it has become the premiere tool for disk data recovery and magnetic mass storage drive maintenance. Written in assembly language, SpinRite still performs as well on a clunky old 4.77 megahertz PC/XT as on a screaming 333 megahertz Pentium II.

While SpinRite 6.0 is running, you can toggle through seven displays:

Purchase and Download SpinRite

More Reading

• PCNet File Catch - SpinRite 6.0
• SpinRite 6.0 for Linux Users
• Anticipate Drive Problems Early with SpinRite v6.0
• Black Viper - Windows XP Super Tweaks
• Black Viper - Optimize your Services
• Advanced Windows Shell Tutorial
• Batch Files (Scripts) in Windows
• File System Heirarchy
• Steve Gibson discussing Defragmenting

Windows Optimization – Intense Part II originally appeared on AskApache.com