ClaudeBot, GPTBot, and AI Crawlers: Why Sites Block Them

ClaudeBot is Anthropic's web crawler. It collects public pages that may be used to train Claude models, and it identifies itself with the ClaudeBot token in its user agent. GPTBot is OpenAI's equivalent. Both vendors say their crawlers obey robots.txt, and many sites use it to shut them out. When we checked the robots.txt files of 73 large sites, 44 blocked ClaudeBot from the homepage and 30 blocked GPTBot.

Sites block these AI crawlers because they take content for model training and send very little traffic back. Some publishers also block them to keep licensing leverage, and some because crawling costs them real infrastructure money. Blocking the training crawler does not have to remove a site from AI search. Anthropic and OpenAI each run three separate bots, and robots.txt can treat them differently.

This guide covers what each bot does, what our robots.txt survey and edge tests showed, how to verify that a request claiming to be ClaudeBot really comes from Anthropic, and how to block exactly the bots you mean to.

What ClaudeBot is, and Anthropic's other two bots

Anthropic's crawler documentation lists three user agents, each with a different job:

• ClaudeBot collects web content that could contribute to model training. Blocking it tells Anthropic to leave the site's future material out of training data.
• Claude-SearchBot indexes pages to improve the quality of Claude's search results. Blocking it can reduce your visibility in those results.
• Claude-User fetches a page when a person asks Claude something that needs it. Blocking it stops those retrievals.

OpenAI's bots page uses the same three-way split, plus an ads checker. Other AI companies publish similar tokens:

Two details catch people out. First, Google-Extended never shows up in your logs. Google crawls with its normal crawlers and reads the Google-Extended group to decide whether that content can be used for Gemini. Second, the old Anthropic tokens anthropic-ai and Claude-Web are not on Anthropic's current list, but 34 of the 73 sites we checked still name anthropic-ai. GeeksforGeeks blocks anthropic-ai and has its ClaudeBot and Claude-Web rules commented out, so the crawler Anthropic actually runs is allowed in. If your file blocks only the old names, it may not do what you think.

OpenAI also says that if a site allows both GPTBot and OAI-SearchBot, it may use the results of one crawl for both purposes. Anthropic's article says its bots honor the non-standard Crawl-delay directive, respect anti-circumvention technologies, and don't bypass CAPTCHAs.

ClaudeBot vs GPTBot: what 73 robots.txt files show

We fetched /robots.txt from 82 high-traffic domains on October 29, 2026: news sites, retailers, social platforms, reference sites, developer docs, and three US government sites. Nine didn't return a readable file to our survey bot's user agent, giving six 403s, one 418, one 202, and one timeout, so we excluded them. For the remaining 73, we asked Protego, the robots.txt parser Scrapy uses, whether each token could fetch the homepage.

The headline numbers:

Naming a token isn't the same as blocking it. A site can name GPTBot in an allow group, and a site can block ClaudeBot without ever writing its name. Three patterns stood out.

ClaudeBot is blocked where GPTBot isn't, never the reverse. Fourteen sites block ClaudeBot and allow GPTBot: The Guardian, The Washington Post, WSJ, the FT, The Atlantic, The New Yorker, Wired, The Verge, Ars Technica, Vox, El País, Le Monde, Tumblr, and Allrecipes. No site in the sample blocks GPTBot while allowing ClaudeBot. Several of these publishers have announced content deals with OpenAI. Condé Nast, which owns The New Yorker, Wired, and Ars Technica, signed one in August 2024. robots.txt doesn't say why a rule exists, but the pattern fits a file that lets the paying partner in and keeps everyone else out.

Allowlists block bots they never mention. WSJ's file starts with User-agent: * and Disallow: /, then lists the crawlers it lets in: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, and some social preview bots. ClaudeBot doesn't appear in the file at all, and it's blocked anyway. Four sites in the sample block ClaudeBot this way, so grepping a robots.txt for "ClaudeBot" won't tell you whether it's allowed.

Training is blocked more often than search. Eighteen sites block ClaudeBot but allow Claude-SearchBot, including the BBC, Bloomberg, Nature, Medium, eBay, and Airbnb. Fifteen block GPTBot but allow OAI-SearchBot. These sites want to appear in AI search answers without supplying training data. The split costs them only a few extra robots.txt lines.

Why sites block AI crawlers

The robots.txt files show what sites do. The reasons come from three places.

Little traffic comes back. A search engine crawls your page and sends visitors back. A training crawler takes the page and, in most cases, sends nobody. Cloudflare measured this as a crawl-to-referral ratio. In July 2025 data, Anthropic crawled about 38,000 pages for every visit it referred, OpenAI about 1,091, and Perplexity about 195. Cloudflare also found that training made up 79% of AI crawling that month. The ratios change from month to month, so read them as orders of magnitude rather than exact figures.

Crawling costs money. Crawlers that read the long tail of a site miss caches and hit origin servers. The Wikimedia Foundation reported that bots make up about 35% of its pageviews but at least 65% of its most expensive traffic. It also reported that bandwidth for multimedia downloads grew 50% from January 2024. Smaller sites on metered hosting feel the same thing on their bills.

Blocking is a negotiating position. The ClaudeBot and GPTBot asymmetry above is the clearest sign of this. A publisher that licenses content to one AI company has a reason to block the others. Their access is what the licensee pays for.

Keep in mind what robots.txt can't do. Anthropic's wording is that blocking ClaudeBot keeps a site's future material out of training data. The rule says nothing about pages that were already crawled. Rules also apply per host. Anthropic says to add the rule to each subdomain, and OpenAI says changes for search take about 24 hours to take effect.

How to verify a ClaudeBot or GPTBot request

Any client can send ClaudeBot in its user agent, so the user agent alone tells you nothing about where a request came from. The source IP does. That matters before you write firewall rules or draw conclusions from your logs.

Googlebot can be verified with reverse DNS. 66.249.66.1 resolves to crawl-66-249-66-1.googlebot.com. That doesn't work for AI crawlers. Addresses we checked in Anthropic's 216.73.216.0/22 range have no PTR record, and WHOIS shows them as an AWS block registered to Anthropic, PBC. The single addresses on Anthropic's list resolve to generic Google Cloud names such as 222.230.162.34.bc.googleusercontent.com. GPTBot addresses in 132.196.86.0/24 also have no PTR record. Anthropic tells site owners to check the source IP against https://claude.com/crawling/bots.json, and OpenAI publishes a JSON file per bot.

The lists we downloaded on October 29, 2026:

Anthropic's single file covers ClaudeBot, Claude-SearchBot, and Claude-User. An IP match proves a request came from Anthropic, but it doesn't prove which of the three bots sent it.

Claude-User doesn't always come from Anthropic's IPs

Not every request with a Claude token comes from Anthropic's servers. Claude Code's WebFetch tool runs on the user's machine, which our Claude Agent SDK guide covers in detail. When we fetched httpbin.org with it from our build server, the request arrived with this user agent:

Claude-User (claude-code/2.1.310; +https://support.anthropic.com/)

It came from our server's own Hetzner datacenter IP, which isn't in bots.json. A request like that is a person's tool fetching a page, and IP verification will mark it as unverified even though nothing was spoofed. How you treat that traffic is a policy decision. An IP check doesn't settle it.

A log check you can run

This script reads a log in the standard combined format, finds requests from the main AI crawlers, and checks each IP against the vendor's current list. It uses only the Python standard library.

"""Count AI crawler hits in an access log and check each IP against the vendor's published ranges."""
import ipaddress
import json
import re
import sys
import urllib.request
from collections import Counter

# Token in the User-Agent -> the vendor's published IP list.
# Claude-SearchBot comes first so it is matched before the shorter tokens.
BOTS = {
    "Claude-SearchBot": "https://claude.com/crawling/bots.json",
    "Claude-User": "https://claude.com/crawling/bots.json",
    "ClaudeBot": "https://claude.com/crawling/bots.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}

# Combined log format: IP ident user [time] "request" status bytes "referer" "user-agent"
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')


def load_ranges(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ai-bot-log-check/1.0"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        data = json.load(resp)
    return [
        ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in data["prefixes"]
    ]


def main(path):
    ranges = {url: load_ranges(url) for url in set(BOTS.values())}
    hits = Counter()
    with open(path, encoding="utf-8", errors="replace") as log:
        for line in log:
            m = LINE.match(line)
            if not m:
                continue
            ip, status, ua = m.groups()
            bot = next((b for b in BOTS if b.lower() in ua.lower()), None)
            if bot is None:
                continue
            try:
                addr = ipaddress.ip_address(ip)
            except ValueError:  # hostname or other non-IP first field
                continue
            verified = any(addr in net for net in ranges[BOTS[bot]])
            hits[(bot, "verified" if verified else "UNVERIFIED", status)] += 1

    for (bot, check, status), n in sorted(hits.items()):
        print(f"{bot:17} {check:10} {status}  {n}")


if __name__ == "__main__":
    main(sys.argv[1])

Run it as python3 verify_ai_bots.py /var/log/nginx/access.log. On a six-line test log with two real ClaudeBot IPs, one ClaudeBot user agent from a documentation IP, a GPTBot IP, a Claude Code fetch, and a Googlebot line, it printed:

Claude-User       UNVERIFIED 200  1
ClaudeBot         UNVERIFIED 403  1
ClaudeBot         verified   200  2
GPTBot            verified   200  1

The Googlebot line is skipped because it isn't in BOTS. If your site sits behind a CDN or load balancer, the first field in the log is the proxy's IP, not the client's. Configure nginx's realip module, or log the CDN's client-IP header, before you trust the results. To identify the other user agents in the same log, paste them into the user agent checker.

How to block ClaudeBot, GPTBot, or both

For most sites the goal is narrower than "block AI". You may want to keep content out of training sets and still show up in AI search and user fetches. This robots.txt does that:

# Training crawlers and training-control tokens
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Claude-SearchBot, Claude-User,
# OAI-SearchBot, and ChatGPT-User
User-agent: *
Disallow: /account/
Disallow: /cart/

Applebot-Extended works like Google-Extended. It controls whether Apple uses content for training and doesn't affect Apple's search features. Test the file before deploying it. With Protego installed (pip install protego), this prints each bot's access to a normal page and a private one:

from protego import Protego

with open("robots.txt") as f:
    rp = Protego.parse(f.read())

for bot in ["ClaudeBot", "Claude-SearchBot", "Claude-User",
            "GPTBot", "OAI-SearchBot", "ChatGPT-User", "Google-Extended"]:
    print(f"{bot:17} /blog/post: {rp.can_fetch('https://example.com/blog/post', bot)!s:5}  "
          f"/account/: {rp.can_fetch('https://example.com/account/', bot)}")

ClaudeBot         /blog/post: False  /account/: False
Claude-SearchBot  /blog/post: True   /account/: False
Claude-User       /blog/post: True   /account/: False
GPTBot            /blog/post: False  /account/: False
OAI-SearchBot     /blog/post: True   /account/: False
ChatGPT-User      /blog/post: True   /account/: False
Google-Extended   /blog/post: False  /account/: False

A named group replaces the * rules

Under RFC 9309, a crawler obeys the one group that matches its name most specifically and ignores User-agent: *. Suppose you have User-agent: * with Disallow: /account/, then add a GPTBot group with only Disallow: /premium/. Protego confirms that GPTBot is then allowed to fetch /account/. If you give a bot its own group, copy into it every * rule that should still apply.

The other direction matters too. If you'd rather slow ClaudeBot down than block it, Anthropic supports Crawl-delay: 1 in the ClaudeBot group. Google ignores Crawl-delay, so don't expect the same line to work for Googlebot.

Enforce at the edge, but keep robots.txt readable

robots.txt is a request. Compliant crawlers follow it, and nothing forces anyone else to. If you want a hard block, return 403 at the web server or CDN. This nginx config blocks the training crawlers by user agent and leaves /robots.txt reachable:

map $http_user_agent $ai_training_bot {
    default        0;
    ~*claudebot    1;
    ~*gptbot       1;
    ~*ccbot        1;
}

server {
    listen 80;
    root /usr/share/nginx/html;

    # Keep robots.txt readable so compliant bots still see the opt-out
    location = /robots.txt { }

    location / {
        if ($ai_training_bot) {
            return 403;
        }
        try_files $uri $uri/ =404;
    }
}

In an nginx:alpine container, ClaudeBot and GPTBot user agents got 403 on / and 200 on /robots.txt. Claude-SearchBot and ChatGPT-User got 200 on both. Keeping robots.txt open matters, because Anthropic warns that IP-level blocks can stop its bot from reading robots.txt at all, so your opt-out may not register. A user-agent rule also blocks anyone spoofing the token, which is usually what you want. To exempt only verified bots, combine the map with nginx's geo module and the vendor IP lists.

What large sites actually enforce

We sent one homepage request per user agent to six sites from a datacenter IP in Germany. These were the vendors' user-agent formats from a non-vendor IP, so any site that verifies IPs would treat every bot row as spoofed.

The edge rules only partly match the robots.txt files. The Guardian's edge matches its file exactly: all three Claude bots get a 403 and the OpenAI bots get through. The New Yorker's robots.txt blocks Claude-SearchBot, but its CDN let that user agent through. CNN's robots.txt blocks all six bots, and its edge returned a 451 with "Content Unavailable For Legal Reasons" to four of them, but it let both search bots through. The BBC and Amazon block several of these bots in robots.txt and enforce nothing at the edge for these user agents. If you see a 403 like these from your own client, the HTTP 403 guide covers the other common causes.

What blocking AI crawlers costs you

Each bot you block has a different cost, which is why the three-way split matters:

• Blocking ClaudeBot or GPTBot keeps future content out of training data. According to Anthropic's and OpenAI's docs, it doesn't remove you from Claude or ChatGPT search, because those use Claude-SearchBot and OAI-SearchBot.
• Blocking Claude-SearchBot or OAI-SearchBot can drop you from AI search answers. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
• Blocking Claude-User or ChatGPT-User stops the assistant from reading your page when a user asks about it. The two vendors document this differently. Anthropic treats Claude-User as controllable through robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a user started the action.
• Blocking Google-Extended doesn't affect inclusion or ranking in Google Search, according to Google's crawler documentation.

For a publisher that sells content, blocking everything except paying partners can be rational. A SaaS company that wants to be recommended in AI answers usually wants the search and user bots allowed.

If you run a crawler or AI agent

The same data applies from the other side.

Don't borrow a ClaudeBot or GPTBot user agent. The table above shows sites return 403 or 451 to those tokens specifically, so a borrowed token gets you blocked more often than a plain one. Sites that verify IPs will also flag you as spoofing. Use your own bot name with a contact URL, as described in the Scrapy user agent guide, and read robots.txt for your own token. Don't assume an allow rule for GPTBot covers you. The guide to crawling without getting blocked covers pacing and the other habits that keep a named crawler welcome. The data scraping legality guide covers where terms of service and copyright come in.

Proxies have a narrow, legitimate role here. If you run a site, test your bot rules from outside your own network. Send a spoofed ClaudeBot request from a residential proxy to confirm your IP verification rejects it. Repeat the check from the countries you serve, since CDN rules can differ by region. CNN's edge set country and city cookies on every response in our test. If you run an agent, proxies for AI agents covers when a stable or local IP helps. Proxies are not a way past a block aimed at your bot. If a site has refused your crawler, rotating IPs hides who is collecting and removes the site's ability to opt out.

FAQ

What is ClaudeBot?

ClaudeBot is the web crawler Anthropic uses to collect public pages that may be used to train its Claude models. It identifies itself with ClaudeBot in the user agent. Anthropic says it follows robots.txt and supports Crawl-delay.

Is ClaudeBot legitimate or a spoofed bot?

The real ClaudeBot comes from IP ranges listed in https://claude.com/crawling/bots.json. Reverse DNS won't confirm it, because those IPs have no Anthropic hostname to check. If a request says ClaudeBot but its IP isn't on the list, treat it as not coming from Anthropic.

Does blocking ClaudeBot remove my site from Claude?

No. Blocking ClaudeBot opts future content out of training. Claude-SearchBot handles search indexing and Claude-User handles user-requested fetches, and each has its own robots.txt group. Anthropic doesn't say that blocking ClaudeBot removes content it already crawled.

What's the difference between ClaudeBot and GPTBot?

They do the same job for different companies: ClaudeBot collects training data for Anthropic and GPTBot for OpenAI. In practice, sites treat them differently. In our 73-site sample, 44 block ClaudeBot and 30 block GPTBot. Fourteen block only ClaudeBot, and none block only GPTBot.

Why is ClaudeBot crawling my site so much?

Training crawlers read deep pages that rarely get human traffic, so they miss caches more often than visitors do. Add Crawl-delay to the ClaudeBot group to slow it down, or Disallow: / to stop it. Then check that the heavy traffic is really from Anthropic's IPs, since any scraper can borrow the name.

Do I need to block anthropic-ai and Claude-Web?

Those tokens aren't on Anthropic's current crawler list. Keeping them in your file does no harm, but they don't replace a ClaudeBot rule. Check that ClaudeBot itself is blocked or allowed the way you intend.

Final thoughts

ClaudeBot, GPTBot, and the other AI crawlers are easy to block. The harder part is blocking the right ones. Each vendor runs separate bots for training, search, and user fetches. robots.txt groups override the * rules, Anthropic publishes one IP list for all three of its bots, and a Claude-User request can come from a laptop rather than Anthropic. Decide which uses you want to allow, write named groups for those bots, test the file with a real parser, and verify IPs before you trust a log entry that claims to be ClaudeBot.