Decide which AI crawlers may read your site. Pick bots, get a robots.txt for the ones that obey it and an .htaccess block for the ones that do not, then test any path against any bot. Training crawlers, AI search and assistants are listed separately, because blocking AI search can also remove you from AI answers.
Matching follows RFC 9309 the way Google does it: the bot uses the most specific User-agent group (or *), the longest matching rule wins, Allow wins a tie, and * / $ work as wildcards.
robots.txt is a request, not a lock: well-behaved crawlers follow it, others need the .htaccess rule. Test the block with curl -A "GPTBot" -I https://example.com/, which should answer 403. Need more? Try the .htaccess generator.
Every night a swarm of crawlers walks your site end to end, and most of them aren't search engines. They're feeding training sets. Some ask nicely and honor robots.txt. Some don't. This tool writes both layers: the polite request, and the locked door for the ones that ignore it.
How to use it
Pick a policy: block every AI crawler, block training but allow AI search (so you still show up when someone asks an assistant a question), or hand-pick bots from the list. The tool writes a robots.txt section and a matching .htaccess block side by side. Copy, download, done. It all runs in your browser.
Each bot in the list shows who runs it, what it's for (training, search, or user-triggered fetches) and whether it's known to respect robots.txt. That last column is the whole reason the second layer exists.
Layer one: robots.txt
User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Bytespider Disallow: /
Note the trick: several User-agent lines sharing one rule group. And Google-Extended isn't a crawler at all; it's a token Googlebot checks to decide whether your pages can train Gemini. Blocking it does not touch your Google rankings. Same idea for Applebot-Extended.
Layer two: the door
robots.txt is a sign on the lawn. For bots that walk past it, Apache answers with a 403 before WordPress or PHP even wakes up:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Amazonbot|PerplexityBot) [NC]
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule ^ - [F]
</IfModule>
The robots.txt exception matters. Let them read the rules; it's how the honest ones learn to stop asking.
The honest limits
User-Agent strings are self-reported. A scraper that wants your content will claim to be Chrome, and no header rule on earth stops that. What this does stop is the large, named, well-behaved fleets, which is most of the volume. For the rest you need behavior: rate limits, and reading your own logs. Drop a day of access log into the Apache log analyzer and look at who's really hitting you.
FAQ
Will blocking AI bots hurt my SEO?
Not if you leave Googlebot and Bingbot alone. The training tokens (Google-Extended, Applebot-Extended) are separate from the search crawlers. Blocking AI search bots like OAI-SearchBot or PerplexityBot does mean you won't be cited in those answers. Your call; the middle policy exists for exactly that.
How do I know if a bot is fake?
Reverse DNS. A real Googlebot IP resolves to *.googlebot.com and that name resolves back to the same IP. Several AI vendors also publish their IP ranges as JSON. Anything claiming to be a big bot from a random VPS is lying.
Where does the .htaccess block go?
Near the top, before your CMS rules. Test it in the .htaccess tester with the User-Agent set to GPTBot and watch it hit the 403.