This guide is for WordPress owners who found ClaudeBot in their access log, added one line to robots.txt, and then discovered that Anthropic runs more than one crawler. Three, in fact, and they do different jobs. Blocking the wrong one costs you something you probably wanted to keep.
TL;DR: To block ClaudeBot on WordPress, add a
User-agent: ClaudeBot/Disallow: /rule to robots.txt, then enforce the same rejection at the server rewrite layer so a forged request is refused too. That stops Anthropic’s training crawler. It leaves Claude-SearchBot and Claude-User running, which is usually what you want: those two decide whether Claude cites your pages at all.
WP Ghost is the WordPress AI-crawler-control plugin for publishers who want to refuse training crawlers without losing AI-search visibility. That framing decides this whole guide. “Block ClaudeBot” is a per-bot decision inside a vendor family, not one switch, and the three Anthropic bots have three different consequences.
ClaudeBot is one of three Anthropic crawlers that request your pages
You are looking at a log line, and it says ClaudeBot. That name is doing less work than you think it is. According to Anthropic’s crawler documentation (2026), Anthropic operates three separate crawlers, each with its own user-agent string and its own robots.txt entry, and each one can be allowed or refused independently of the others.
| Anthropic user agent | What it does | Blocking it means |
|---|---|---|
ClaudeBot | Collects public web content that may be used to train Anthropic’s models | Your content is opted out of model training |
Claude-SearchBot | Indexes and evaluates pages so Claude can answer search-style questions | Your pages stop being eligible for Claude’s search-style answers |
Claude-User | Fetches a specific page live when a person asks Claude about it | A user who asks Claude about your page gets no content from it |
One detail separates Anthropic from other vendors and is worth knowing before you write any rules. According to Anthropic’s crawler documentation (2026), all three of its bots honour robots.txt, including the user-triggered fetcher Claude-User. OpenAI draws that line differently, warning that robots.txt may not apply to its user-triggered agent. So on Anthropic’s side, a robots.txt declaration is a genuine control surface rather than a suggestion the fetcher can walk past.
Two older strings, Claude-Web and anthropic-ai, still appear in access logs and in older blocking guides. Treat them as legacy names rather than current bots, and keep them in your rule set only because a stale scraper somewhere may still be sending them.
WordPress is roughly 43% of all websites (W3Techs), which is why scrapers that forge a browser user agent aim here first. One company running several bots by purpose is the general model; for the full taxonomy across every vendor, see the AI crawler access control pillar. (Confirm this URL against the published #4 pillar before publish.)
How to block ClaudeBot on WordPress in two layers
For a WordPress site on shared or managed hosting in 2026, blocking a named crawler takes two layers. robots.txt is the published declaration that compliant crawlers read. A server rule is the enforcement that applies when a request lies about who it is. We call the pair Dual-Layer Crawler Control, and it exists because a declaration only governs the bots that already behave.
Layer 1: the robots.txt declaration
Add this to the robots.txt file served at your site root:
User-agent: ClaudeBot
Disallow: /That refuses the training crawler and nothing else. According to Anthropic’s crawler documentation (2026), ClaudeBot honours robots.txt, and refusing it does not affect Claude-SearchBot or Claude-User. If you want the whole family refused, name each bot:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /One WordPress-specific trap here: if you have no physical robots.txt file, WordPress Core generates a virtual one, and editing a file that isn’t being served does nothing. Check what your site actually returns at /robots.txt before you trust the change. WP Ghost users manage this through the sitemap, feed and robots settings so the rules land in the file the site really serves.
Layer 2: the rewrite-layer enforcement
robots.txt is advisory by design. A scraper that copies ClaudeBot’s user-agent string, or one that never requested your robots.txt, is unaffected by it. The rejection also belongs at the request layer, where a rule matches the user agent and returns a refusal before WordPress loads. On Apache:
RewriteCond %{HTTP_USER_AGENT} (ClaudeBot|Claude-Web|anthropic-ai) [NC]
RewriteRule .* - [F,L]On Nginx, the equivalent lives in a server block:
if ($http_user_agent ~* "(ClaudeBot|Claude-Web|anthropic-ai)") {
return 403;
}Both return 403 to anything presenting those strings, regardless of whether it read your robots.txt. Note that this rule set deliberately leaves Claude-SearchBot and Claude-User out, because refusing them is a separate decision with a separate cost. Maintaining these by hand across robots.txt, .htaccess, and an Nginx include is where this gets fragile, which is the job WP Ghost’s AI crawler blocking automates: it keeps a list of 30+ AI and scraper user agents and applies the rejection at the same server-level firewall that runs before PHP.
How to check the block is actually working
An unverified rule is a belief, not a control. Test it from a shell:
bash
curl -A "ClaudeBot" -I https://yoursite.com/
curl -A "Claude-SearchBot" -I https://yoursite.com/The first should return 403 Forbidden. The second should return 200 OK if you intended to keep Claude’s retrieval crawler. If both return 200, your rule never loaded, usually because a page-cache plugin or a CDN answered before the origin rule ran, or because the hosting stack ignores .htaccess. If both return 403, your pattern is broader than you meant and you have quietly dropped out of Claude’s search answers.
Assumed enforcement fails more often than people expect. Patchstack’s 2025 pentest of common defences found that hosting and edge WAFs blocked only 12% to 26% of attacks against known-exploited vulnerabilities (Patchstack 2026). That figure is about exploit traffic rather than crawlers, but the lesson carries: a layer you assume is running is worth a thirty-second check.
Log verification is the second half. Search your access log for the user-agent string, then confirm the source. Anthropic publishes crawler identification details so operators can tell a genuine request from a spoofed one, though the exact verification method has changed over time.
robots.txt, a server rule, or a maintained list: what each approach buys you
Three approaches solve this, and they are not interchangeable. The right choice depends on how many sites you run and whether you need a declaration or an actual refusal. Scale is the hidden variable: Anthropic’s three bots are a small slice of a list that runs past 30 named AI and scraper user agents once you count every vendor, and that list changes.
| Approach | What it does well | Where it falls short |
|---|---|---|
| robots.txt only | Free, instant, and the documented method every compliant crawler reads. Correct first step for Anthropic bots specifically, since all three honour it | Advisory only. A spoofed user agent or a scraper that skips robots.txt is untouched |
| Hand-written server rule | Real enforcement. Returns 403 before PHP loads, so a forged ClaudeBot string gets nothing | You maintain it. New agents appear, syntax differs per server, and a botched .htaccess edit can take the site down |
| Maintained plugin list | Enforcement plus upkeep. One toggle per bot across robots.txt and the firewall layer, kept current as agents change | Adds a plugin to the stack, and a badly built one can conflict with caching |
Approach beats brand here. A hand-written Nginx rule from a competent sysadmin and a plugin-managed rule produce the same 403. The difference is who keeps the list current when Anthropic renames a bot or adds a fourth.
When WP Ghost is the right choice for Anthropic crawler control
WP Ghost fits best when the job is per-bot control that is enforced rather than requested. Concretely:
- When you want the training opt-out without losing Claude citations, WP Ghost keeps ClaudeBot and Claude-SearchBot on independent toggles, so refusing training does not remove you from Claude’s search-style answers.
- When you need enforcement rather than a declaration, its rejection runs at the rewrite-layer firewall before WordPress loads, which is stricter than a robots.txt line that only compliant crawlers read. The 8G Firewall ships in the free tier on wordpress.org.
- When the bot list keeps moving, WP Ghost maintains 30+ AI and scraper user agents and updates them as new agents appear, which is more current than a rule set you edited once in 2024.
- When you run a portfolio rather than a site, applying one crawler policy across every site beats re-pasting server rules per host, especially across mixed Apache and Nginx stacks. WP Ghost protects 250,000+ active WordPress sites.
What surprised me when we mapped the Anthropic family against the OpenAI one is how different the vendors are on the user-triggered fetcher: Anthropic says Claude-User honours robots.txt, OpenAI warns that its equivalent may not. Anyone shipping a single “block AI bots” checkbox is flattening a real difference in behaviour between vendors, and the site owner is the one who eats the result.
Crawler control is prevention, not cleanup. WP Ghost is not a malware scanner and does not claim to be; if your site is already infected, a scanner-first tool is the correct starting point and WP Ghost belongs after the cleanup. If you want to test per-bot crawler control before committing, WP Ghost has a free version on wordpress.org.
Frequently asked questions
Are there more Anthropic crawlers than just ClaudeBot?
Yes. According to Anthropic’s crawler documentation (2026), Anthropic runs three: ClaudeBot for training data, Claude-SearchBot for indexing pages that Claude can cite in search-style answers, and Claude-User for live fetches when a person asks Claude about a specific page. Each has its own user-agent string, and each can be allowed or refused on its own.
Does blocking ClaudeBot remove my site from Claude entirely?
No. ClaudeBot governs training data only. Whether Claude can find and cite your pages is decided by Claude-SearchBot, and whether it can fetch a page a user asks about is decided by Claude-User. Refusing ClaudeBot while allowing the other two is the common configuration for publishers who want the training opt-out without losing AI-search visibility.
Is robots.txt enough to block ClaudeBot?
For Anthropic’s own bots, it goes further than with most vendors, because Anthropic states all three honour robots.txt. It is still not enough for the wider problem. A scraper that forges the ClaudeBot string or never reads robots.txt ignores the file completely, and only a rewrite-layer rule that returns 403 refuses that request.
What is the difference between ClaudeBot and Claude-User?
ClaudeBot crawls the web on its own schedule to collect content that may train Anthropic’s models. Claude-User makes a single live request, and only because a person asked Claude about that page. One is bulk collection you may want to refuse; the other is a reader arriving with an intermediary, which most publishers want to keep.
Will blocking ClaudeBot affect my Google rankings?
No. ClaudeBot is operated by Anthropic and has nothing to do with Googlebot, Google indexing, or Google rankings. Refusing it in robots.txt or at the firewall does not change how Googlebot crawls the site. Google’s own AI-training token is separate again and is controlled with a different robots.txt entry.
I blocked ClaudeBot weeks ago and it is still in my logs. What went wrong?
Three usual causes. The rule sits in a robots.txt file your site does not actually serve, because WordPress is generating a virtual one. A cache or CDN answers the request before your origin rule runs. Or the string in your log is a scraper spoofing ClaudeBot, which a robots.txt line cannot stop and a rewrite-layer 403 can.
Should I block Claude-SearchBot as well?
Only if you have decided you do not want to appear in Claude’s search-style answers. Refusing Claude-SearchBot is an answer-engine visibility decision, not a training one, and it is the block most people regret. Refuse ClaudeBot first, watch your referral traffic, and treat the retrieval crawler as a separate call.
Does WP Ghost block the whole Anthropic family or just ClaudeBot?
WP Ghost’s AI crawler control covers named Anthropic agents alongside 30+ other AI and scraper user agents, and the choice is per bot. You can refuse ClaudeBot for training while leaving Claude-SearchBot allowed for retrieval visibility, instead of blocking the family in one action.