Updated robots.txt for WordPress
Implementing an effective SEO robots.txt file for WordPress will help your blog to rank higher in Search Engines, receive higher paying relevant Ads, and increase your blog traffic. Get a search robots point of view... Sweet!
robots.txt
WordPress robots.txt SEO
WordPress robots.txt file can make a huge impact on your WordPress blogs traffic and search engine rank. This is an SEO optimized robots.txt file.Alternate robots.txt files with Htaccess
So here's the basic idea: There are 2 sites, a development site and a live site. They are essentially mirrors of each other in terms of they have the same files. You need to disallow all search engine robots from indexing and crawling the development site, while allowing full crawling of your live site. Htaccess to the rescue!
Robots.txt Secrets From Matt Cutts
The secrets in this post were really more of enlightening bits of seo wisdom. The secret is how to combine robots.txt with meta robots tags to control pagerank, juice, whatever.
Blocking Bad Bots and Scrapers with .htaccess
Want to block a bad robot or web scraper using .htaccess files? Here are 2 methods that illustrate blocking 436 various user-agents.
SEO Secrets of AskApache Part 2
This is part II of the Advanced SEO used on AskApache.com Series and describes how to control which urls are indexed by Search Engines and how to move them higher up in Search Results.
AskApache Password Protection, For WordPress
AskApache Password Protect adds some serious password protection to your WordPress Blog. Not only does it protect your wp-admin directory, but also your wp-includes, wp-content, plugins, etc. plugins as well. Imagine a HUGE brick wall protecting your frail .php scripts from the endless attacks of automated web robots and password-guessing exploit-serving scripts.
Allow AdSense Crawler Access
Google AdSense uses the robots.txt file to optimize Ads displayed on your site for the Mediapartners-Google robotAdvanced HTTP Redirection
Learn about the 7 different HTTP response codes specifically reserved for redirection. 301, 302, 303, 304, 305, and 307.
Htaccess SetEnvIf and SetEnvIfNoCase Examples
SetEnv, SetEnvIf, and SetEnvIfNoCase directives conditionally set environment variables accessible by scripts and apache based on HTTP Headers, Variables, and Request information.
SEO in WordPress
Search Engine Optimization for WordPressInstruct Search Engines to come back to site after you finish working on it
Nifty SEO tip to get Search Engine Bots to check your site every hour until you finish working on it and tell them you are finished.
Optimize a Website for Speed, Security, and Easy Management
Learn how to setup, configure, secure, optimize, and create a low-maintenance website the AskApache way. I'm piecing together all the hacks, tricks, methods, and ideas discussed throughout this blog and all across Netdom and glueing them all together to show you how to have the most optimized, crazy fastest, and best website setup I can think of.
WordPress .htaccess in-depth
WordPress uses a file named .htaccess to rewrite all requests to the main index.php file. This article explains what the htaccess file rules look like and what they actually do.
Protecting Files with Advanced Mod_Rewrite Anti-Hotlinking
If you have files on your site that you don't want indexed by malicious search engines, grabbed and leeched by malicious spammers, or stolen and made available elsewhere, you can use mod_rewrite to drastically reduce or totally reduce that activity.
Fight Blog Spam with Apache
Fighting Blog Spam with Apache htaccess and other methods.
Mod_Rewrite Basic Examples
mod_rewrite is very useful in many situations. Yet some behaviors were not so obvious when I started to mess with it. After many testings, I understand it much better, now. Having said that, I do not pretend to know it perfectly. I also make mistakes.
Wget Trick to Download from Restrictive Sites
Before
After trick
I am often logged in to my servers via SSH, and I need to download a file like a WordPress plugin. I've noticed many sites now employ a means of blocking robots like wget from accessing their files. Most of the time they use .htaccess to do this. So a permanent workaround has wget mimick a normal browser.
THE Ultimate Htaccess
.htaccess is a very ancient configuration file for web servers, and is one of the most powerful configuration files most webmasters will ever come across. This htaccess guide shows off the very best of the best htaccess tricks and code snippets from hackers and server administrators.
You've come to the right place if you are looking to acquire mad skills for using .htaccess files!
Mod_Security .htaccess tricks
Mod_Security rivals Mod_Rewrite in the amount of features it provides. I decided to go ahead and post what I learned about it today, even though its tough to give away such awesome htaccess and apache tricks.. Learn how to control spam once and for all, conditionally log/deny/allow/redirect requests based on IP, username, etc.. Mod_Security is so fine!
Password Protection Plugin Status
Enumerating Permissions can be Annoying
Don’t ask me how because I won’t tell you, but on one of the hosts I was testing on that did not allow direct access I was able to get the Apache server running as dhapache to erroneously write a file into my users blog directory. This is a big security no-no and I now have my .htaccess file written into the blog directory where it should go, but instead of my php script’s user having write access to the file so I can modify it, its owned by dhapache! Because the file is owned by dhapache I shouldn’t even be allowed to know it exists, but there it is. So the next step was to try and take ownership of the .htaccess file so that I could modify it. I tried and tried but was unsuccessful, I couldn’t modify it so that was another dead end. Actually it took me awhile to figure out how to remove the file from my directory. Being that it was owned by dhapache I couldn’t delete or modify it using my php process or even through ftp/ssh! Sysadmins regularly run find commands that search the servers for any files owned by dhapache that should not be there as this is a big red flag that someone has found a way to manipulate dhapache which could potentially lead to modifying dhapache-owned server config files, which sometimes is all it takes to hack your website and server.. Luckily I was able to delete it by basically running the hack again to overwrite the file.
PHP to handle HTTP Status Codes for ErrorDocument
Fast, HTTP Protocol, protection. If you are reading this article, you already know enough about the benefits of making sure your site can handle HTTP Protocol Errors. This is a nice single php file with no dependencies or requirements, will work on anything. Optimized for minimizing bandwidth and resource-hogging connections from bots and spambots.
<?php
ob_start();
@set_time_limit(5);
@ini_set('memory_limit', '64M');
@ini_set('display_errors', 'Off');
error_reporting(0);
SEO Secrets of AskApache.com
Learn how in a year, with no previous blogging experience this blog was able to rank so high in search engines and achieve 15,000 unique visitors every day. Uses combination of tricks and tips from throughout AskApache.com for Search Engine Optimization.
Dealing with Mobile Visitors using Bad Browsers
Actual Htaccess Files from My Server
#### No https except to wp-admin -
# If the request is empty ( implies fopen or normal file access by a php script )
RewriteCond %{THE_REQUEST} ^$ [OR]
# OR if the request if for wp-admin or wp-login.php
RewriteCond %{REQUEST_URI} ^/(wp-admin|wp-login.php).*$ [NC,OR]
# OR if the Referer is https
RewriteCond %{HTTP_REFERER} ^https://www.askapache.com/.*$ [NC]
# THEN skip the following rule, basically all this does is force https or badhost to be redirected
# BUT because of the above 3 rewritecond's, this won't break poorly written admin scripts
RewriteRule .* - [S=1]
RewriteCond %{HTTPS} =on [OR]
RewriteCond %{HTTP_HOST} !^www.askapache.com$ [NC]
RewriteRule .* https://www.askapache.com%{REQUEST_URI} [R=301,L]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9} /(wp-admin/.*|wp-login.php.*) HTTP/ [NC]
RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R=301,L]
Redirect index.php to root
WordPress blogs show the same duplicate content for https://www.askapache.com/index.php and https://www.askapache.com/. If you've read about using a robots.txt file for WordPress SEO, than you already understand this setup results in Duplicate Content penalties being levied against your Blog and Web Site by Search Engines.
Introducing the Online Figlet ASCII Generator
Look at the text "askapache". That's pretty boring huh? It would be cool to have a nerdy textual representation of that for extra nerdy stuff like styling my /robots.txt file, email list signatures, forum sigs, etc. But who has time to create that by hand? If only there were an online tool to create it..
Bam.
Advanced Google Analytics 404 Error Page
Advanced 404 Not Found Error page usage with Google Analytics. This makes it very easy to diagnose and keep track of 404 Error messages from within google analytics. Enjoy!
Speed Up Sites with htaccess Caching
2 awesome ways to implement caching on your website using Apache .htaccess or httpd.conf. Both methods are extremely simple to set up and will dramatically speed up your site!PirateBay and Anonymous SOPA Press Releases
SOPA: Anonymous Lists Their Demands
A rallying cry on the occassion of the Web's first mass blackout
As we watch the web go dark today in protest against the SOPA/PIPA censorship bills, let's take a moment and reflect on why this fight is so important. We may have learned that free speech is what makes America great, or instinctively resist attempts at silencing our voices. But these are abstract principles, divorced from the real world and our daily lives.
Free speech is the foundation of a free society. We can have the vote all we want. We can donate money wherever we want. But unless we're able to talk to each other and figure out collectively _what_ we want, those things don't matter.
We believe a healthy society doesn't allow its artists, musicians and other creators to starve. The copyright industry has been justly criticized for abusing the political process in a desperate attempt to maintain its role as a cultural gatekeeper, a business model made obsolete by a digital age of free copies. But the RIAA, MPAA & IFPI deserve our opprobrium for making enormous profits while often leaving the very artists it claims to represent *poorer* than they would be as independents.[1] While the public may have greater access to the few artists deemed sufficiently marketable to gain mass media promotion, fewer and fewer of us are making art and music in our own lives.
We call upon all freedom loving Internauts to join us. We further call upon our legislators, bureaucrats and the media & telecommunications industries to immediately begin implementing our demands. The future of free speech is bright, and clear - either stand with us or get out of the way.
PirateBay Press Release regarding SOPA...
So, the whole basis of this industry, that today is screaming about losing control over immaterial rights, is that they circumvented immaterial rights. They copied (or put in their terminology: "stole") other peoples creative works, without paying for it. They did it in order to make a huge profit. Today, they're all successful and most of the studios are on the Fortune 500 list of the richest companies in the world. Congratulations - it's all based on being able to re-use other peoples creative works. And today they hold the rights to what other people create. If you want to get something released, you have to abide to their rules. The ones they created after circumventing other peoples rules.
The reason they are always complainting about "pirates" today is simple. We've done what they did. We circumvented the rules they created and created our own. We crushed their monopoly by giving people something more efficient. We allow people to have direct communication between eachother, circumventing the profitable middle man, that in some cases take over 107% of the profits (yes, you pay to work for them). It's all based on the fact that we're competition. We've proven that their existance in their current form is no longer needed. We're just better than they are.
Htaccess Rewrites for Moving Urls
Links to htaccess tutorials and articles
Links to htaccess tutorials and howtos in the htaccess forum
Ultimate Htaccess Part II
Here is even more information from the Ultimate Htaccess Part I. For now this is very rough and you will want to come back later to read it.
Internet Census 2012 – Port scanning using insecure embedded devices – Carna Botnet
While playing around with the Nmap Scripting Engine (NSE) we discovered an amazing number of open embedded devices on the Internet. Many of them are based on Linux and allow login to standard BusyBox with empty or default credentials. We used these devices to build a distributed port scanner to scan all IPv4 addresses. These scans include service probes for the most common ports, ICMP ping, reverse DNS and SYN scans. We analyzed some of the data to get an estimation of the IP address usage.
Running a Reverse Proxy in Apache
Fast and Easy Custom WordPress New User Registration
Registering on a WP blog requires entering a username and email then checking your email for an auto password.
Too much work to put your users through?
This uses just email and password and does it all.
Advanced WordPress wp-config.php Tweaks
The bottom line for this article is that I want to make WordPress as fast, secure, and easy to install, run, and manage because I am using it more and more for client production sites, I will work for days in order to solve an issue so that I never have to spend time on that issue again. Time is money in this industry and that is ultimately (time) what there is to gain by tweaking WordPress.
Note: I spent no time on readability, this is primarily a read the code and figure it out article.. This is for advanced users looking for a reference or discussion and for those of you looking to advance. Feedback would be great if you make it that far..
An AskApache Plugin Upgrade to Rule them All
So my blog as been rather quiet for almost a year now, and very few updates if any have been released for my Password Protection PLugin, my Google 404 Plugin, and definately not for my AskApache CrazyCache plugin, which I will be releasing last... So for all of you who've helped me out by sending me suggestions and notifying me of errors and sticking with it... Just wanted to say sorry about that, and thanks for all the great ideas.. Well, I've been sticking with it as well believe it our not. I manage to get free days once in a while, and then its time to jam.