Comments for AskApache searching on robots.txt

Comment on WordPress robots.txt SEO by jimmy choo

jimmy choo — Mon, 05 Jul 2010 09:02:08 +0000

thanks a lot for this useful article

Comment on WordPress robots.txt SEO by HDxxx

HDxxx — Sun, 04 Jul 2010 15:19:05 +0000

Thanks, This is exactly what I needed.

Comment on Speed Up Sites with htaccess Caching by How do I decrease my site size?

How do I decrease my site size? — Wed, 02 Jun 2010 14:46:56 +0000

[...] can, enable gzip compression on your server. Yahoo (search gzip components), Apache – Make sure you set up caching (search cache-control) – Minimize stuff on the page. – If you can, disable any Apache [...]

Comment on Real-Life Htaccess Files from My Server by Mezanul

Mezanul — Sun, 30 May 2010 19:25:09 +0000

Very helpful for me. Thanks for sharing your knowledge with us and helping us. Thanks a lot :)

Waiting for the rest of the useful htaccess stuff! :)

Comment on Updated robots.txt for WordPress by Spunky Jones

Spunky Jones — Sat, 29 May 2010 07:23:13 +0000

I am using the following in my robots.txt file on one of my directories. I noticed that I have over 1,700 pages and only 400 pages indexed by Google. My question is, which statements in my robots.txt file are causing the pages not to get index? Furthermore, do I actually need any of this in the robots.txt file?

#  PARTIAL access (Googlebot)
User-agent: Googlebot
Disallow: /*.php$
Disallow: /*?
Disallow: /*?*
Disallow: /?s=A&p=1
Disallow: /*/trackback/

Comment on Real-Life Htaccess Files from My Server by mcp

mcp — Sat, 22 May 2010 09:46:41 +0000

hi there,
i am agree with that My day job is with a school district and I ran across your stuff when we had some issues. I’m still reading and learning so I can get better with these files and lock things down better. Thanks for sharing this knowledge.

Comment on SEO with Robots.txt by SEO in Wordpress

SEO in Wordpress — Mon, 17 May 2010 08:22:03 +0000

[...] robots.txt usage for SEO boost Use XHTML Phrase elements for SEO boost [...]

Comment on WordPress robots.txt SEO by evilripper

evilripper — Mon, 10 May 2010 09:47:08 +0000

thanks!
i had some problem with comment-page-1 title duplicates and maybe I solved with these lines:

Disallow: /comment-page-*/
Disallow: /blog/comment-page/

Comment on Optimize a Website for Speed, Security, and Easy Management by kirti

kirti — Tue, 04 May 2010 11:50:14 +0000

Nice web development articles.. Thanks

Comment on Updated robots.txt for WordPress by Alan

Alan — Mon, 03 May 2010 01:09:57 +0000

Hi again,

I commented a few days back but it didn’t seem to take so here goes again… :-)

I understand that WP has a virtual robots.txt file. The xml sitemap plugin has an option to add the Sitemap to the virtual robots file. I have chosen that option and the virtual robots.txt file is recognized by Google Webmaster Tools and does in fact include the Sitemap. The sitemap plugin states the following:

The virtual robots.txt generated by WordPress is used. A real robots.txt file must NOT exist in the blog directory!

So, what to do? Is the correct answer to not choose the option via the xml sitemap plugin to use the virtual robots.txt file and to create a new actual robots.txt file, where I can also include the Sitemap?

Also, why include the Allow option in the robots.txt file for individual Google bots? Why not just Allow all (*) ? And why use any “Allow” command at all? Google will crawl the site providing that there is no Disallow command.

I look forward to your reply. Thanks!

Comment on Instruct Search Engines to come back to site after you finish working on it by Sawa

Sawa — Sat, 01 May 2010 05:34:43 +0000

Please add useful html code for 503 response…

Comment on Optimize a Website for Speed, Security, and Easy Management by Yman

Yman — Tue, 27 Apr 2010 11:39:52 +0000

Good stuff. Great website management tips!

Comment on Real-Life Htaccess Files from My Server by Tony Dunsworth

Tony Dunsworth — Tue, 20 Apr 2010 16:02:07 +0000

Been following your site for a while now. My day job is with a school district and I ran across your stuff when we had some issues. I’m still reading and learning so I can get better with these files and lock things down better.

I would love to see the whole file so I can keep learning more about tightening things down even further.

Love the site.

Tony

Comment on Instruct Search Engines to come back to site after you finish working on it by best seo

best seo — Tue, 20 Apr 2010 01:37:40 +0000

yeap, this is a nice trick, didn’t know it :) however, at a certain moment some pages have to be notified as dead to SEs. I can’t figure out if getting lots of 503 is more acceptable to google than 404, maybe you can explain.

Comment on Real-Life Htaccess Files from My Server by Apache mod rewrite help needed - IDN Forums - Internationalised Domain Names

Apache mod rewrite help needed - IDN Forums - Internationalised Domain Names — Mon, 19 Apr 2010 02:37:49 +0000

[...] Yes, you can do it with htacces. Check this example, which blocks based on keywords like Viagra: http://www.askapache.com/htaccess/re…tml#menu0-el10 One of my favorite htacces ever. __________________ Join the IDN Domains Group on Facebook | [...]