If you use a robots.txt file and AdSense, make sure you give the Google AdSense robot Mediapartners-Google full access.
Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler.
If your site is using a robots.txt file, the AdSense crawler may be blocked from crawling your webpages. Therefore, we may not be able to serve you the most relevant ads based on the content of your website.
Google AdSense uses the robots.txt file to learn what content is on your site in order to display the most relevant and optimized ads.
To grant the Google AdSense crawler access to your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:
User-agent: Mediapartners-Google* Disallow:
This change will allow the AdSense bot, Mediapartners-Google to crawl the content of your site and provide you with the most relevant Google ads for your pages.
For all other AdSense information visit the Google AdSense Help Page.
See the Updated WordPress robots.txt file