Home  »  Google  »  Allow AdSense Crawler Access

by 8 comments

If you use a robots.txt file and AdSense, make sure you give the Google AdSense robot Mediapartners-Google full access.

What Google Says

Use robots.txt

Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.

Do not restrict AdSense crawler

If your site is using a robots.txt file, the AdSense crawler may be blocked from crawling your webpages. Therefore, we may not be able to serve you the most relevant ads based on the content of your website.

Allow AdSense Crawler Access

Google AdSense uses the robots.txt file to learn what content is on your site in order to display the most relevant and optimized ads.

To grant the Google AdSense crawler access to your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*

This change will allow the AdSense bot, Mediapartners-Google to crawl the content of your site and provide you with the most relevant Google ads for your pages.

For all other AdSense information visit the Google AdSense Help Page.

See the Updated WordPress robots.txt file


July 3rd, 2007

Comments Welcome

  • Eric F
    User-agent: Mediapartners-Google*
  • jehzlau

    oh great! thanks for sharing! :)

  • rbreaker

    thx now it's work :)

  • June

    I'm running a classfieds ads site. Based platform is wordpress + classipress. Do I have to put up any crawler on my site.

    What the is the username and password should i provide? Is it an admin password or the host server
    user name and password like the one I used when doing FTP.

  • Navin Kunwar

    Its asking for user name and passord. should i create separate account for that ?

  • Kalpana Chauhan

    can you please tell me the difference betweeen User-agent: Mediapartners-Google and User-agent: Mediapartners-Google*

  • mmorden

    Does anyone know how to allow the Google Crawler to access AdSense on my website? I use Godaddy as my host. Please can someone help me?

  • Maxwell Purrington

    It is very grand to tell us that: "Simply add the following two lines to the top of your robots.txt file:" but for those of us (which I suspect is the great majority of bloggers) what is the value of this information if we are not told where the "robots.texfile" is located?

    Can't step by step instructions be provided with the assumption that the reader is a computer novice?

Related Articles

My Online Tools
Popular Articles

Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman


It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain