Home » Google » Allow AdSense Crawler Access

Allow AdSense Crawler Access

by Charles Torvalds

If you use a robots.txt file and AdSense, make sure you give the Google AdSense robot Mediapartners-Google full access.

What Google Says

Use robots.txt

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.

Do not restrict AdSense crawler

If your site is using a robots.txt file, the AdSense crawler may be blocked from crawling your webpages. Therefore, we may not be able to serve you the most relevant ads based on the content of your website.

Allow AdSense Crawler Access

Google AdSense uses the robots.txt file to learn what content is on your site in order to display the most relevant and optimized ads.

To grant the Google AdSense crawler access to your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*

This change will allow the AdSense bot, Mediapartners-Google to crawl the content of your site and provide you with the most relevant Google ads for your pages.

For all other AdSense information visit the Google AdSense Help Page.

See the Updated WordPress robots.txt file


Comments Welcome

Information is freedom. Freedom is non-negotiable. So please feel free to modify, copy, republish, sell, or use anything on this site in any way at any time ;)

My Online Tools

Popular Articles
Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two.
-- Richard M. Stallman

It's very simple - you read the protocol and write the code. -Bill Joy


Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | htaccess file |

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service | @Htaccess

↑ TOPMain