This is part 2 of the exhaustive Htaccess Tutorial. I realized it was so lengthy that search engines and visitors were having real problems with it, so I moved half of it here. But this gave me the opportunity to add a ton of new stuff that I hadn't been able to add to the main htaccess tutorial. And now this new part 2 is already twice as big as the original! To sum up, this is a work in progress.

Moving Half of Original Tutorial to a new URL - SEO Rewriting

Well, the original tutorial was at /htaccess/apache-htaccess.html, but it was such a huge article that search engines were dropping it! So I split it into 3 new urls.

1. /htaccess/htaccess.html - 1
2. /htaccess/htaccess-rewrites.html - Rewrites
3. /htaccess/apache-htaccess-2.html - Part 2

The /htaccess/apache-htaccess-2.html is a lot of unfinished new stuff, and the /htaccess/htaccess-rewrites.html (this very page) still needs to be split several times to get the filesize down.

301 Link Juice

The /apache-htaccess.html url has been around since 2006, and it was full of original ideas and examples for using htaccess. Almost every htaccess guide or tutorial published since then has many of the examples and ideas from that tutorial, (I encourage people to modify and republish everything on this site according to copyright). I know it, the authors sometimes know it, but the main point is Google for sure knows it. That's the beauty of creating unique content, Google sees that. So that /apache-htaccess.html link has extreme link juice, from all the sites, books, papers, and presentations that have linked to it since 2006.

Filesize Is important

I like to use my blog as a way to keep notes about my research organized, and I am extremely good at doing research, unfortunately, that means I have a huge article. The filesize for the html alone is larger than all the other resources like javascript and images, combined.

• So that means it is very difficult to view on a mobile device, or a slow connection.
• For google-bot and other search engine crawlers and robots, this is a huge problem (I made a big mistake letting it get that size).
• The robot has to parse an enormous single html file, containing hundreds/thousands of external and internal links, and its such an issue they can decide to just skip indexing that url until the filesize is manageable.
• Once again, it's all about the human experience, a huge single file is not good for anyone who isn't printing it out to read offline.

Starting Fresh without losing juice

So since the page wasn't being indexed much since it was so huge, I decided to split up the content into new separate urls and utilize a 301 Redirect to transfer all the link juice from /apache-htaccess.html to /htaccess.html.

301 Redirect Timeline

1. Now, the idea is to take the first pages from the original multi-page guide, do a little improvement on that content, and save it to the new url /htaccess.html.
2. Then just continue taking the next page from the remaining original guide and creating new pages using the original links structure. /apache-htaccess-2.html, /apache-htaccess-3.html, etc.. This is a secondary backup to the new /htaccess.html url, which will receive the 301 link juice from the old url, but these secondary pages will help keep the links on external sites good.
3. Then, I setup a 301 Redirect in my .htaccess file to redirect the old url to the new /htaccess.html url.
4. Finally, I delete the old url and it is replaced forever by a 301 Redirect pointing to my new location, filesize problems eliminated.

301 Redirect Code Used

This is so easy to do with RedirectMatch, way faster and easier than using mod_rewrite to handle this, and much less overhead. Note that this is a general command that I will leave up for a few weeks and slowly tighten it up by looking at my Google Analytics and Apache Logs. For instance, this first redirectmatch rule is an older RedirectMatch I still have active to redirect all the old links pointing at /2006/htaccess/apache-htaccess.html to /htaccess/apache-htaccess.html from when I ditched the date-based permalinks back in 2007, and you can see it is a little tighter than the 2nd one which also redirects requests for apache-htaccess.html/feed/ or trackback or whatever.

RedirectMatch 301 ^/2006/.*apache-htaccess.html$ http://www.askapache.com/htaccess/apache-htaccess.html
RedirectMatch 301 ^/.*apache-htaccess.html(.*)$ http://www.askapache.com/htaccess/htaccess.html$1

Advanced 301 Redirects for SEO

I wrote a couple of articles that go into detail about maximizing the SEO with linking and redirects, it remains one of my most helpful articles to anyone trying to rank higher the right way, the Google way.. SEO Secrets of AskApache Part 2

Htaccess Rewrites for Moving Urls originally appeared on AskApache.com

I've written alot about how I speed up websites, but there is much more to cover. Here are the 3 ways I speed up my site by playing with the Google Analytics Tracking Code.

1. Host the Google-Analytics ga.js file locally
2. Set Correct Cookie Domain
3. Make Sure Google Analytics Loads

Default Google-Analytics Tracking Code

<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
var pageTracker = _gat._getTracker("UA-2452345-5");
pageTracker._trackPageview();
</script>

1. Hosting the Google ga.js file locally

Instead of calling the http://www.google-analytics.com/ga.js file in your site's html, you can instead save the ga.js source code and serve it from on your own server! This reduces DNS lookups, gives you control over caching, and if you combine the ga.js with a javascript file already on your site like I do, you reduce HTTP Requests, reduce browser cache lookups, etc.

For this site, I added the ga.js code to my http://static.askapache.com/z/j/apache-999.js file, then I compressed it online and saved to serve!

How to Do it: Serve External Javascript Files locally for Increased Speed

2. Set Correct Domain for Cookie

For those websites who employ multiple sub-domains to serve static content, and you are using the default google-analytics javascript code for website statistics, you may be able to get some speed by forcing the google analytics cookie to only be sent for your non-static subdomain.

I just noticed this on my site the other day while watching the wire:

The Google Analytics cookie was being set for the domain .askapache.com and so it was being sent needlessly with every request to every subdomain, when it only should be sent for requests to www.askapache.com.

Specify Google-Analytics Cookie Domain

So I googled for an answer for awhile and finally found the solution in the ga.js code itself. There is a command called _setDomainName that lets you set the domain for the cookie. The following code sets the GA cookie for .www.askapache.com, ensuring that the cookie will not be sent needlessly for images and other static content.

var pageTracker = _gat._getTracker("UA-732153-7");
pageTracker._setDomainName("www.askapache.com");
pageTracker._trackPageview();

3. Make Sure Google Analytics Loads

Look at the source code for this page to see how I do it. Here is what is going on:

1. Add the google javascript at the bottom of my source code to make sure the HTML and DOM is fully loaded before first calling the script for download.
2. Check that the google-javascript is loaded by testing that the object called _gat exists, which is the internal object created.
3. Initiate google-analytics

The Safer Google-Analytics Include Code

<script type="text/javascript">
if (typeof(_gat) == "object") {
  var pageTracker = _gat._getTracker("UA-732153-7");
  pageTracker._setDomainName("www.askapache.com");
  pageTracker._initData();
  pageTracker._trackPageview();
}
</script>

Top 3 Speed Tips for Sites using Google Analytics originally appeared on AskApache.com

One of the ways I speed up AskApache.com is by downloading all the external javascript files to my server and then serve them from my own server instead of the external site. Currently I'm using this method to serve the Google Analytics javascript file, and the Quantcast javascript file.

Speed Benefit

The speed benefit occurs because normally a site visitor has to perform a DNS query for both the google-analytics.com and quantcast.com servers, create a connection to each server, and then download the files. By hosting these scripts on your own server you remove all the extra DNS queries, and by hosting all the files on your single server you enable advanced HTTP 1/1 protocol features like Pipelining the requests, which means a single connection for multiple files.

Cache and Compression Control

Serving these javascript files from your own server gives you complete control over the caching and compression settings of those files, which may not seem like a big deal... unless you know about the advanced caching and compression techniques available to make your site super-fast.

Caching Control

By using the mod_rewrite cache hack like I do, you can make sure that a fresh version of the .js file is used by your visitors.

Compression Control

You can also specify advanced compression for these files, which could mean using Apache's mod_deflate or mod_gzip modules, or you can even use a compression tool like Dojo's Shrinksafe, Packer, JSMin, or YUICompressor to further compress the javascript files.

Automated Downloading with Crontab

Using crontab I tell my web server to run this shell script every day, to make sure I have updated versions of all the files. Here's the crontab entry I use to run this shell script every day.

@daily /bin/sh /home/cron-scripts/grab_javascripts.sh 2>&1 &>/dev/null

Shell Script to Download JS

I created a simple Bash shell-script to download the javascript files. All you need to do is modify the variables to save to your directory and send your site as the Referring URL and you are good to go. You can add as many files as you want by adding them to the JSFILE array.

You can also download the shell-script: grab_javascripts.sh

#!/bin/bash
# 2008-09-30
umask 022
VER="2.0"
 
### Set as the referring url for downloads
SITE=http://www.askapache.com/
 
### Directory to save the downloaded files
JSD=$HOME/j/
 
### The files to download
JSFILE[0]=http://www.google-analytics.com/ga.js
 
### run script with args to turn on debugging
[[ $# -ne 0 ]] && set -o xtrace && set -o verbose
 
### SHELL OPTIONS
set +o noclobber # allowed to clobber files
set +o noglob # globbing on
set -e # abort on first error
shopt -s extglob
 
#--=--=--=--=--=--=--=--=--=--=--#
# pt
#==-==-==-==-==-==-==-==-==-==-==#
function pt(){
  case ${1:-d} in
   i) echo -e "${C6}=> ${C4}${2} ${C0}"; ;;
   m) echo -en "\n\n${C2}>>> ${C4}${2} ${C0}\n\n"; ;;
   *) echo -e "\n${C8} DONE ${C0} \n"; ;;
  esac
}
 
#=# CATCH SCRIPT KILLED BY USER
trap 'kill -9 $$' SIGHUP SIGINT SIGTERM
 
#=# MAKE MAIN SCRIPT NICE AS POSSIBLE SINCE IT DOESNT DO MUCH
renice 19 -p $$ &>/dev/null
 
#=# TURNS ON COLORING ONLY FOR TERMS THAT CAN SUPPORT IT
C="\033[";C0=;C1=;C2=;C3=;C4=;C5=;C6=;C7=;C8=;C9=;
  C0="${C}0m";C1="${C}1;30m";C2="${C}1;32m";C3="${C}0;32m";C4="${C}1;37m"
  C5="${C}0;36m";C6="${C}1;35m";C7="${C}0;37m";C8="${C}30;42m";C9="${C}1;36m";
clear
echo -e "${C1} __________________________________________________________________________ "
echo -e "| ${C2}             ___       __    ___                 __             ${C1}         |"
echo -e "| ${C2}            / _ | ___ / /__ / _ | ___  ___ _____/ /  ___        ${C1}         |"
echo -e "| ${C2}           / __ |(_-</  '_// __ |/ _ \/ _ \`/ __/ _ \/ -_)       ${C1}         |"
echo -e "| ${C3}          /_/ |_/___/_/\_\/_/ |_/ .__/\_,_/\__/_//_/\__/        ${C1}         |"
echo -e "| ${C3}                               /_/                              ${C1}         |"
echo -e "|                                                                          |"
echo -e "|                 ${C4} LOCAL JAVASCRIPT FILES SCRIPT Version ${VER} ${C1}              |"
echo -e "${C1} __________________________________________________________________________ ${C0} \n\n"
 
#=# BUILD INCOMING DIRS
[[ ! -d "${JSD}" ]] && pt m "BUILDING DIRS" && mkdir -p $JSD &>/dev/null && pt i "CREATED $JSD" && pt
 
#=# DOWNLOAD JAVASCRIPT FILES
cd $JSD && pt m "DOWNLOADING JAVASCRIPT FILES"
for theurl in "${JSFILE[@]}"; do
pt i "${theurl}"
curl -m 60 --connect-timeout 10 --retry 10 --retry-delay 180 -s -S -L -e '${SITE}' -A 'Mozilla/5.0' -O "${theurl}"
done
pt && cd $OLDPWD
 
######## curl options
# -S Show error. With -s, make curl show errors when they occur
# -s Silent mode. Don't output anything
# -e  Referer URL (H)
# -H <line> Custom header to pass to server (H)
# -L  Follow Location: hints (H)
# -A <string> User-Agent to send to server (H)
# -m <seconds> Maximum time allowed for the transfer
# --connect-timeout <seconds> Maximum time allowed for connection
# --globoff  Disable URL sequences and ranges using {} and []
# -O  Write output to a file named as the remote file
# --retry <num>  Retry request <num> times if transient problems occur
# --retry-delay <seconds> When retrying, wait this many seconds between each
# --retry-max-time <seconds> Retry only within this period
#############################################################################
 
exit 0

Serve External Javascript Files locally for Increased Speed originally appeared on AskApache.com

The goal of AskApache.com is simple - To provide free access to knowledge and data with the goal of empowering people.. or more melodramatically: "Power to the People!"

Why the name AskApache? AskApache was chosen to show and pay respect to the contributors of the Apache Web Server. Literally it means to ask Apache when facing a problem, by searching the Open-Source, contacting a board/list, or browsing the documentation.

The Author

I work for a multimedia production / brand development & marketing company here in Indianapolis, the greatest city in the world! I started this blog in Dec., '06 to familiarize myself with WordPress as a blogging platform among other things for my job. I help expand/create a growing companies market-share online (and offline) by managing and building the brand and developing high-quality leads and loyal customers by building online funnels fed by SEO traffic and on/offline marketing campaigns. Other than the design and development costs for our minimum 12 month contract, we operate using a gain-share formula where we receive a percentage of the increased revenue that results from our services. This translates to fanatical attention to improving the bottom-line, and focus on long-term sustainable growth and overall success.

It's important to me to note that unlike most systems set up like this, we are absolutely interested in creating customers/leads/whatever by providing real value for them. It's much more difficult and takes a lot longer, but it is also self-sustaining as value attracts in an exponential manner. We are likely different than any model you've seen before in terms of lead-sales-marketing types.

My Background

I started on DOS, then Windows 2.11 (released 1989), and learned BASIC for my first language... then IBM PC assembly followed by Java, which I used as an excuse to stop learning assembly.. And the visual studio suite I used for several years around win95 era. Hearing so much about linux I went out and immersed myself in Red Hat, which had great documentation and books. Then came the BSD's.. The first computer I had access to was DOS but not windows yet, so thats why I learned the BASIC languages. I liked DOS so much I got some DOS POWER TOOLS books from PC Mag and was able to do whatever. DOS was like my gameboy or xbox, and knowing it so well has really come in handy since a form of DOS is still to this day in Windows, it's kinda like having cheat codes.

C and C++ are the only languages I ever took a class on in school and I know them decent enough. The thing is, I've spent my whole computer life reverse-engineering everything I can get ahold of, and so I read source code more than I would care to say. Almost all of it is C, perl, or more unusual stuff like yacc, bison, or some type of scripting. Sometimes I get my best ideas from reading source code and am always looking for new more optimized ways to get something done in the worlds best code. I read it like a book more than code because I've just been doing that my whole life. Nobody said hey here's a computer, this is called DOS, and here's how you do this:.. I literally didn't even know what a computer did, so I learned DOS at first by traversing the whole dir-tree and trying to open everything. The worst part is that unlike any linux windows doesn't have any kind of man pages, so I learned by doing something and seeing what happened! It's amazing what you can learn by viewing every file included with an operating system, and that was my start.

I've used javascript for maybe 10 years.. CSS and xhtml I've learned in great detail since 03. Flash, actionscript, pretty much anything ever released by Adobe/Macromedia I've been using since 2000. PERL was the first main language I learned when I switched to linux, I also code in PYTHON, RUBY, and OCAML, but now I use PHP more than anything for web dev.

The only formal programming classes I've taken was a semester of C in a high school AP class back in the day. BASIC and QBASIC was a game to me, found it by accident. At that point I was maybe around 8 or so, and I went to the public library and got a book on Assembly called (it's currently sitting on my bookshelf) "Assembly language primer for the IBM PC and XT". I remember going through it page by page and taking detailed notes and following along with the book. Definately the hardest thing I had ever tried to learn in my life, but I stuck with it for a whole summer since I didn't know any better, and wrote a few simple programs from within DOS. I'm convinced that experience of trying to learn assembly before even knowing much about ocmputers is the reason why I find languages so intuitive, nothing else I code is even close to being as difficult as learning assembly, which is about as low-level as you can get, its been around for over half a century.

So I'm definately not even close to being as good as the best coders of the world and their various specialty languages.. But on the other hand I am totally flexible attaining the best possible solution, so it's like the Bruce Lee method of having no method, and this lets me be extremely flexible when going about trying to find the best solution. I look at the whole spectrum from the network level to the host and app level and my experience at reverse-engineering security, networks, and software gives me a very different way of looking at things. Coders are taught to plan and build from start to finish. I have always done the exact opposite. Once I know the finish point it's a done deal, I will be able to get there no matter what and the code is usually much smaller and leaner doing it that way.. Or maybe I'm just dyslexic or something like that.

Of all the areas of study relating to computers, I was most fascinated with networking and communication, and wanted to reverse-engineer everything to really understand it. Somehow I figured out how to use the hyperterm of windows 3x and hooked the computer up like the fax machine was and all of a sudden my world became much much bigger as I discovered the web, arpanet,gopher repos, telnet servers, etc..

I frequented BBS's regularly before the "Internet Browser" became available with mosaic. At that time before HTTP was standardized it was like, 100% text only, no presentation or style.. it was all about the content. I taught myself how to use terminal and a modem to wardial (calling phone numbers sequentially searching for a computer) based on that old War Games movie, and it actually worked. The first machine I got access to in this way was a Sun machine used by a library. That was my first network experience and within 5 years me and a couple friends had gained access to machines all over the world, including the Indianapolis International Airport Ticket Terminals and the Air Traffic Control itself! But then HTTP and Mosaic exploded on the scene and were soon followed by the great censor AOL. After the novelty wore off and everyone used AOL, that's when today's Internet was born in my opinion. AOL censored the net and was really it's own network, and that is what people got used to thinking about the Internet in terms of what it was.

Security Focus

Most people like me who were lucky enough to discover the Internet (gopher, ftp, etc.) at the time when you could only surf the web using text-based tools were also lucky to learn alot of great security information that was freely shared all over the world by various hackers and groups.

Security Path

From those early days on I was heavily involved in the security research scene, mostly networking security, then server/application security. I know quite a bit about the malicious activities one hears about from time to time, but other than learning how they work and how to execute them (against a test system mostly) I stayed away from cracking.

Of course back in those days I would occasionally create a virus or trojan but just for fun and not for exportation. Does anyone else remember all those automatic virus creation tools that came out once Windows 95 was released? Haha those old progs rock!

Working for an ISP

Around the turn of the Millenium I worked for a small Internet Service Provider. I learned how to wire buildings, houses, and the networking devices required by the ISP to connect to the backbone and customers. My boss's previous job was for the US Military, and all he could tell me about his work there was that it was top-secret cryptography work. I learned alot about OS security and Network Security while working there.

The coolest thing about working for an ISP was learning about the Networking and Administrative capabilities and requirements specific to ISP's.. I learned how to setup, operate, debug, and admin switches, hubs, routers, servers, backup systems, ISDN connects, etc..

Unix - Linux - BSD

Working at the ISP opened my eyes to the power of BSD/Unix, and I've spent almost a decade using hundreds of linux distro's and all BSD flavors. My favorite Linux distro's are Arch Linux, BackTrack 4, Gentoo, Ubuntu, and Slackware, in that order. I love debian. I run OpenBSD 24/7 but I also always like to have 1 computer running FreeBSD for "fun."

Education Background

For a longgg time I was planning on going to school for an advanced network security degree, available at only a couple of colleges in the whole country. So I took all the networking/computer courses available at my college, and I even managed to convince the Department Head to allow me to skip all pre-requisites for any CIS courses and enrolled as a freshmen in the advanced classes.

I was immediately frustrated with the curriculum, none of it was challenging or new... it was just the same boring path towards a degree :(. So I kept the books, ditched class for a couple years, and went into business for myself doing freelance custom security audits for organizations and interested parties. That allowed me to spend my time hacking and educating myself, and paid the rent in my one-bedroom apartment.

Employed as a Security auditor/analyst

For about 5 years I helped small but very privacy-conscious organizations identify exploits in their operations and policies.. Basically I hacked until I got in or got fired.. A couple times I got root the same day I started the initial recon, but those were ALL Windows machines run by amateurs so it doesn't really count..

Most of the time it would take several weeks or months of mapping.. probing.. reading.. reading.. reading.. and packet-crafting before I had to give up on what I think of as the Technical phase of an operation. I was unsuccessful many, many, times at this phase, but rarely unsuccessful after the second phase, which is where I would utilize sneaky social engineering.

One technique that worked extremely well was to spoof an email from some service company used by the target organization to an employee's email account and immediately call that employee and walk them through opening the email and unknowingly installing my customized trojan. The customized trojans were really devious and really really cool. These were the precursors to the IRC botnet virii. Once I had the trojan installed I could generally get to any computer on the network within a day or two.

Areas I worked in

While I was a security consultant I was basically a Jack-of-all-trades kind of resource. The audits I performed were incredibly lengthy and all were done remotely. Meaning that I wasn't going to visit your organization and run diagnostics and security scanners on your internal network, I would literally insist that my clients only provide me with a public facing node, mostly a domain/server address.

This allowed me to perform the audit the same way that a real hacker would. And which was a methodology I had used for several years and was comfortable with. The goal with every audit was to systematically look for vulnerabilities or misconfigurations that when exploited would provide me with root access to the internal/external network of the organization.

Most of the exploits that I used to gain access were application-level vulnerabilities like sendmail, ssh, phpBB, shopping carts, etc.. Nothing very advanced, just a lot of research and testing.

Once I achieved access (a few times I was totally unsuccessful) and outlined how I did it and/or how to fix the vulnerabilities I was basically done. More often than not my clients would be pretty happy about my services and would ask me to do other misc. work.

I was involved in securing and locking down many web applications like phpBB, Joomla, Shopping carts, etc.. and also was hired to setup or re-configure servers and software to be more secure.

Alot of my clients would want hire me to provide information about someone. I would be enlisted to trace and locate various entities, such as:

• The source of a DDOS attack against a website (potentially one hired by a competitor)
• A spammer masquerading as my client
• A cracker selling stolen credit card numbers or other merchandise on the internet black market
• A person spreading lies or talking bad about my client on the Web

I became pretty proficient at being able to identify, find, and locate anyone using the Internet for communication. For example, many times a client would locate some private information that wasn't supposed to be public posted anonymously on some forum or message board or email. I was very successful at this, and I don't think I ever failed to trace the owner of an email account. This activity got me involved with honeypots and honeynets, among other tracking tech.

Leaving Security to Pursue Web Development

So I was all fired up to get my Masters degree from the best school in the country and that degree would guarantee me a good job in a field I absolutely love, but around 2006 I switched.

Government, Public Sector, Private Sector Security Ambitions

Ok so in the security world, everybody knows that Governments have the best toys, the secret technology, carte blanche for its employees, basically everything on my "Top 10" list. But most people also know that the Government doesn't pay very well and there are some personal security concerns that crop up when working for a Superpower. I disdain money..To me its an invention someone came up with a long time ago, and I won't let the economic vultures steal my dreams.

After graduation I was interested in working for the US Air Force, which has some of the highest tech in the world, or for an intelligence-related government agency. I am not that into cryptography, so the NSA didn't look like a good fit. Today there are dozens of Government computer security agencies and groups, and most of the major nations are building the armies of the future cyber wars today and have been for years. Some big players: China, U.S., Soviet-Union Countries, N. Korea, Italy, France, Japan, and Germany.

Exact Moment I switched to Web Development

Of course security means constant never-ending research, and I had been researching the possible ways of exploiting a systems security by hijacking the boot-up process. Mostly this requires physical access, such as plugging in a USB drive, CD, serial line, etc.. But I was also learning about the boot process that networking devices like switches and routers execute. Many of these devices automatically look for configuration files, operating-system upgrades, or boot files on their network while booting up, and to make a long subject short by using some network hacking you could talk to these devices and provide them with modified files, effectively taking over the device.

For regular computers I was learning about the various boot loaders used by the different operating systems like Windows, BSD, Linux, etc.. (think grub, lilo, etc.). The exciting thing I learned was that all these OS's that can be booted from a floppy or CD basically can write data directly to specific areas on the actual root drive...

During my research of the various OS's boot process (from when you turn on the machine to when the operating system starts) I learned alot about rootkit technology.. kernel-level rootkits interested me and led me to some very dangerous areas on the net. At this point I was surfing the web from a Linux machine using a VMWare Virtual workstation for extra protection. That machine's internet connection came through a customized OpenBSD Machine that was running Snort for Intrusion Detection and IPTables - a fully stateful firewall. That firewall was connected to the net through an additional Stateful firewall/router. So I was being very very careful to avoid getting cracked.

I won't divulge what exactly I was researching, but I discovered an incredible resource of information on some security topics that I had never before (or since) seen. The day after I found this site my VMware workstation was rooted, my underlying linux operating system was rooted, and my firewall/IDS was rooted. Luckily I had configured an older 3rd machine as a syslog server and cut the TX wires so it was invisible, and thats how I discovered the intrusion. All I will say about it is that the intrusion was so sophisticated and the attackers tracks so well covered that I took out all the hard-drives and put them in storage so I could examine them in a few years with better tools and hopefully I can discover the methods used by the attacker and maybe discover where they came from. Needless to say, I was out of my league and someone was watching me.

Web Development Opportunity

So I had spent all those years learning about the vulnerabilities of various software and hardware, and then learning how to fix and correctly setup and run them. I already knew html, javascript, basic perl/php/shell-scripting, and I was experienced at running and administrating web servers at this point, so when one of my clients called me late on a Friday and begged me to help them build a website due on Monday I decided to give it a shot.

My First Website

OWWW! Thats me howling in pain when I recall that weekend of web design hell. I built the whole site using Microsoft Frontpage, something which should be outlawed, and not knowing much about CSS I built the whole 4 page site using HTML tables.

The site was completed on time and opened my eyes to the endless possibilities of web development. I spent the next year trying to MASTER CSS and best-practice standards-based coding, then I learned more about Ajax and javascript, and also started using server-side programming quite a bit.

My first 10 or so websites were each drastically better than the last, and all of them were created by hand in code without templates of any kind. A good 15+ page site would take me about 3weeks to 5weeks to complete back then, now I build online presence's designed to increase the clients bottom line. This includes everything and anything our client wants and to start we do a 1year contract, it takes a lot of time and effort over a long period to really achieve greater market share (leads) using the web in high-competition markets. I tell the sales/marketing people who work with me to tell the clients that we can do anything, because we pretty much can. I'm just so thrilled at how easy all this web development stuff is compared to the security stuff... It's night and day and I'm really enjoying myself.

Ideal Candidate looking to work with me

1. The right attitudes
2. Willing to challenge and be challenged
3. Scrappy & will do whatever it takes
4. High attention to detail
5. Want to make history not read it
6. You have a history of achievement at whatever level you've been at.
7. You are the best person you know at what you do.
8. You are faster than fast
9. You can be competitive but nice at the same time
10. Skills

If you fit these characteristics then you should definately NOT send your resume as I hate computer science resumes, what a freaking waste of time. I don't care if you are physically and mentally handicapped or even completely just a repulsive individual :) Truly, the only thing that counts with me is skills. . . either technical prowess or non-technical business know-how -- I always enjoy meeting new people in the industry so send me an email and introduce yourself, It's great fun to call up fellow developers in London and Japan... Feels cool so it must be!

Technology Interests

1. CSS
2. XHTML (strict!)
3. Javascript (unobtrusive, strict!)
4. NetSec (network security, hack the planet)
5. PHP, Python, Ruby, Perl, Ocaml, Shell-Scripting (strict mostly!)
6. AJAX (strict,crossbrowser,degrades gracefully!)
7. Linux {Arch-Linux, Slackware, Gentoo, Red Hat, Fedora}
8. BSD/Unix {OpenBSD, FreeBSD, NetBSD}
9. Software {Photoshop, Dreamweaver, Sorenson, Flash, QuickTime, ffmpeg, nirsoft, sysinternals, Adobe Creative Suite, etc.}
10. Windows {for Workgroups, 3.1, 95, 98, me, 2000, NT Workstation&Server, XP Pro&Home SP1&SP2}
11. And of course, if not especially, the Apache Web Server, .htaccess, and mod_rewrite!

AskApache.com Info

1. Google +
2. Popular Web Design and Development Blogs on Google
3. Top Weblogs on Alexa
4. Technorati
5. AskApache Twitter
6. Typekey
7. Stumbleupon
8. Last.fm
9. seomoz
10. LiveJournal
11. Digg
12. Flickr
13. MyBlogLog
14. WordPress Plugins, Support Forum, Codex Wiki
15. BlogCatalog

More AskApache

1. BlogCatalog
2. StatsAholic
3. Alexa
4. Technorati
5. Compete
6. Quantcast
7. DomainTools
8. Live
9. Google
10. FindForward
11. Alexa DirectoryAlexa Info
12. MyBlogLog htaccess Group

AskApache mirrors

Every once in awhile I come across a project that is so indescribably great that I decide to help out by mirroring and contributing to the content.

• cURL / libcurl
• Savannah - nongnu
• WireShark Network God

Google if you are reading this, I'm willing to relocate. ;) Microsoft, no thanks.

About AskApache originally appeared on AskApache.com

First let me say that I don't read SEO stuff, I don't participate in the SEO community, I only have an interest in regards to the technology used by the search engines. All websites are hosted on servers, mostly Apache, and that is the primary topic of this blog. During the past year of this blog, my research into non-seo areas has turned up some very valuable SEO techniques.. All of them legal, ethical, and genuinely good for the Internet at large.

Update: SEO Secrets part II: Advanced Indexing and Pagerank Control

Some Background

I started this blog in late 2006, my first foray into blogging, and I've been extremely successful at achieving top ten google rankings and maintaining on average 15K unique visitors/day (per google analytics) 85% of which come from search engine traffic.

NOTE: I take it for granted that anyone reading AskApache is an expert of some skill, if you aren't I apologize, I can't waste time on the easy stuff.

Prerequisite SEO

There are literally hundreds of thousands of SEO articles on the net, 99.9% of which are absolute garbage. Especially in the sense that they just repeat the same 10 year old stuff. However, to do any kind of advanced SEO like I am going to discuss in this article, I am assuming that you, the intelligent reader, has already read those and has a basic understanding of SEO fundamentals like meta tags, titles, keywords, etc.

First, Great Content

The foremost and most important step in achieving any kind of traffic is to produce great content. I'm sure you've heard that a million times, but let me break it down how I perceive it. Before I even started to mess with SEO for AskApache.com I began by writing articles. At that point I didn't have a clue what my blog was going to be about or even if I was going to be doing it after a week.

What is Content

For me, being a top-paid professional web developer, I spend about 80% of my time doing research. I think that is a bit uncommon, but its a throwback from the 10 years I spent in the network/computer security field, where research is 99% of the job, a story for another time perhaps.

So the research I was doing at that time was about best-practice standards-based web design, mainly XHTML Strict, CSS, and unobtrusive javascript. Each of those subjects has become near and dear to my heart, and each should also be mandatory learning for anyone interested in SEO. The best advice I can give towards that end is checking out the CSS, Javascript, and XHTML Strict source code for this page and site. And of course the holy W3.org.

In addition to striving to master those 3 subjects, I was also and always will be researching web programming languages like PHP, Ajax, Ruby, and Server Technology like Apache. Although I should note that my research into Apache and server technologies is more of a hobby than a job requirement, also a throwback to my days in the security industry and of course my love for open source software.

My Content

So basically I was spending 25% of my time at work actually working, and the other 75% of the time I would research how to do something better, faster, the best. Incredibly, I discovered or re-discovered a ton of tips, tricks, and methods to aid me in my work. I was learning so much valuable information that I joined a couple of forums to discuss them and get feedback on making them even better. Soon I realized that I was one of a small few who actually post content to a forum instead of just questions, so I decided to write my tutorials down on a blog, and AskApache was born.

So that is why this blog is comprised of almost 100% tutorials, and why almost all of them are completely original works you won't find elsewhere. That's how I create content, but you might do something different. Whatever it is that you do for content, just make sure you are providing VALUE with everything you do. Not to everyone, just stuff that you would consider to have value if you were reading it.

Second, Great Site

Ok so I had 10 or so great articles that I knew would provide value for many web developers, but so what? Nobody cares you know.. That's when I decided to take a closer look at the software that was running my new blog, WordPress, and I've been hacking the code ever since on my never-ending quest to be the best and know the most advanced web development. You'll see why in a couple paragraphs.

I Mean, a Really Great Site

By great, I mean you need to make it incredibly user-friendly. Every design and development decision you make should be about the visitor. THATS the number one key to success on the net, regardless of endeavor. Here is a list of things you definately need to have before you do SEO, I'm not listing obvious stuff like descriptions, titles, and good writing.

1. Intuitive and circular, your website should be a spider-web of urls.
2. Easy to read, plenty of white-space, design is your decision but I like minimalistic.
3. Super fast rendering. You need effective caching and optimization.
4. A very helpful 404 error page, hopefully never seen.

Focus in on your URL's

Many sites that use a CMS of some kind, be it Drupal or WordPress, have hundreds or thousands of URL's even if they only have 10 actual posts/articles.

Removing Duplicate Content

You've all heard this before, but almost no-one has taken it to the level I am going to discuss. Bear with me.

Removing duplicate content is actually a very straightforward process if you know what you are doing, and if you don't, well that's why I'm going to quickly explain how to really do a good job.

Locate Duplicate Content and URLS

People misunderstand that you should just not repeat the same paragraph in a different article, that is partially true, but the main impact this has on your site is if you can access the same article from more than a single URL.

I hope you realize you MUST use pretty urls like my site and not codey looking ones with question marks. You can find any potential duplicate urls on wordpress with the rewriterules plugin. Also look at Google's webmaster tools to look for any duplicate urls, and you can use xenus link sleuth tool as well.

Remove Duplicate Urls with .htaccess

Once you've found duplicate urls, you need to instruct google and other search engine robots to be redirected to the correct url. By doing a 301 redirect you tell the search engines NOT to index the bad url, only the good one. Below are some of the .htaccess code I use on this site to accomplish this technique, this is gold I myself use so pay attention. It works.

301 Redirects with mod_rewrite

First lets start with one everyone should know, and the most common, to www or not to www?

RewriteCond %{HTTP_HOST} !^www\.askapache\.com$ [NC]
RewriteRule ^(.*)$ http://www.askapache.com/$1 [R=301,L]

Its a highly rare individual who has seen this one, which forces requests for .html/ to .html

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /[^\.]+\.html/\ HTTP/ [NC]
RewriteRule ^(.*)\.html/$ http://www.askapache.com/$1.html [R=301,L]

301 Redirects without mod_rewrite

For the rest of the duplicate urls that you find, I like to use Redirect and RedirectMatch.

This redirects requests that start with an & or /&amp or /( to my homepage.

RedirectMatch 301 ^/&(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/&amp(.*)$ http://www.askapache.com/
RedirectMatch 301 ^/([\(]+)(.*)$ http://www.askapache.com/

This redirects requests with //whatev to /whatev

RedirectMatch 301 ^//(.*)$ http://www.askapache.com/$1

But this is just a brief look at what you will have to spend some time on. There are detailed guides to doing this with mod_rewrite and using Redirect on my blog. Its time now for some real SEO tips. The heart of the matter, as it were.

Wrap It Up

So I realize that was brief, so I want to really stress 2 things or you won't take away much from part 1.

Locate Duplicate URLS

This is truly one of the most important things in my personal experience. I personally take this as extreme as I can, I regularly grep my access files, mod_security and error log files looking for bad URLS. I am always checking them out to see if someone has a bad link to me somewhere, or if someone just typed it in wrong. If its a bad link on a site, I will very politely attempt contacting the webmaster about it until they fix it.

Even I, with my many colorful years of Internet travel, was caught off-guard by the variety and creativity and the sheer number of urls people are using to link to my site. I found that often bad links would be published because my URL was just too long, so I shortened the URL's. Now of course bad links can't really even touch my site with all my 301's in place.

Besides grepping your server's logs, the 2nd best place to locate duplicate urls or just plain wrong urls is by using Googles free webmaster tools. They keep track of all the bad urls linking to your site and allow you to download this data in a .csv spreadsheet format. The first time I checked into this I found over 1,000 bad links, after a couple months with my RewriteRules and 301 Redirects, I've narrowed the list down to under 50 most months. That is a powerful reason to use 301 Redirects, as we'll really get into in part 2.

301 Redirect Bad URLS

Finding the bad urls takes some time, a couple hours even, and then the whole reason you do that is to be able to create 301 Redirects for all of those bad urls to good urls.

One reason that I wasn't even aware of until several months ago is that when Googlebot locates a bad URL for your site, it tries to access it, and if you haven't planned for this in advance, your page most likely will return a 200 OK status header, or if you are lucky a 404 Not Found error, both of which really hurt you.

Basically, a 200 response will produce duplicate content in 99.9% of the time, and 404 responses will whisper to Google's algorithms that you don't know what you are doing and your site isn't authoritative. 200 means google will index your site, 404 means google won't index your site, but it also won't give up trying for awhile, which takes away from your real urls.

What a 301 Response tells Google

301 Responses were practically invented for user-agents/clients/web-crawling robots like google. They instruct the client, whether that be a persons browser or a googlebot, that the resource/page that they are looking for is actually at a different URL. This is an authoritative response that makes googlebot and other search engines ecstatic because now they can give up on the 200 and 404 responses that didn't really give them an answer either way.

On the other hand, a great 404 can and should be just as powerful as a 301, but hardly anyone uses them in the correct way according to HTTP 1.1 or 1.0 Specifications. We'll tear that subject apart further down the road.

I'll leave this topic for now with one last idea, 301 Redirects when implemented and used correctly, actually redirect the page rank and search engine ranking for itself to the redirected / correct url. That means if you have 1000 unique links pointing to your article, and all of them are incorrect in some way, if you can 301 redirect all of those bad links to your correct link you now have 1000 new good links! It has to be done right and in a classy way though of course.

Now that you have content and a great site, its time to SEO like a mofo.

SEO Secrets of AskApache.com originally appeared on AskApache.com