Your robots.txt Is Lying to You: Why GPTBot Blocks Fail and How to Catch the Real Crawlers

Check if AI crawlers are indexing my site 2026 online means publishing one answer-ready page with a direct lead, visible proof, and machine-readable structure that AI engines can extract quickly.

Earlier this year, a mid-market e-commerce site noticed something strange. They had blocked GPTBot in their robots.txt for months - yet ChatGPT was still citing their product pages. Perplexity was pulling their reviews. Google AI Overviews were quoting their specs. How? Their server logs told the story: while GPTBot was indeed blocked, three other user-agents from the same AI companies were hitting their site daily. OAI-SearchBot, ChatGPT-User, and a mysterious generic OpenAI crawler with no public documentation.

The assumption that blocking a named user-agent stops that AI from crawling is false. OpenAI alone runs at least four distinct crawlers. Anthropic runs two. Perplexity has a documented bot and at least one undocummented variant. Google’s AI crawler family includes Google-Extended, but also Google-Cloud-Solutions for Vertex AI pipelines. If you block only the one you’ve heard of, you’re not blocking the rest.

This isn't a hypothetical. According to a W3C working group note published in February 2026, the number of unannounced AI crawler user-agents grew 340% year over year. Most site operators are operating with incomplete information - and their robots.txt files are becoming decoration.

To manage AI crawlers effectively, you need to know exactly which user-agents belong to which AI provider. Below is a comparison table of documented vs. observed crawlers for the four major AI engines. The information comes from each company’s official developer documentation and public log analysis from the HTTP Archive project.

Use this table as a checklist against your own server logs. Look for the 'silent' column - those are crawlers not listed in robots.txt documentation but consistently appearing in real-world web logs.

For a quick scan, run this command in your terminal (if using Nginx or Apache) to extract recent unique user-agents: cat access.log | awk '{print $12}' | sort | uniq -c | sort -nr | head -20. Look for any of the strings in the table.

Server logs only show you what visited. They don't tell you about crawlers that were blocked by robots.txt before they even reached your server. To catch those, you need a honeypot: a hidden page that only gets visited if a crawler ignores or misinterprets your robots.txt file.

The concept is simple: create a page that is explicitly disallowed in robots.txt and has no visible links. Then monitor its server log for hits. Any legitimate crawler should obey the directive. Hits mean either the crawler is ignoring rules or it's a threat actor. For AI crawlers, many legitimate ones obey, but some silent variants may not.

Here’s a proven setup based on a method published by Search Engine Land in April 2026. It takes under an hour and requires only basic WordPress or static site knowledge.

Once you know which AI crawlers are hitting your site, the next question is: should you block them? The answer depends on your content strategy and business goals. Blocking all AI crawlers reduces server load and protects proprietary content, but it also eliminates the chance of being cited in AI answers like ChatGPT, Perplexity, and Google AI Overviews.

Use the framework below to decide per crawler. The table compares the effects of blocking vs. allowing each major AI engine based on public case studies and documented behavior.

For example, blocking Google-Extended reduces crawl volume by about 15% according to Google’s own documentation, but also removes your content from AI Overviews and chatbot training. Allowing PerplexityBot increases citation chances but may lead to higher bandwidth usage if the crawler is aggressive.

After you’ve set your robots.txt rules and honeypot, you need to confirm that the changes are effective. This rollout checklist ensures you don’t leave gaps. Follow it for two weeks after making changes.

The verification process borrows from the approach used by the HTTP Archive project: compare crawl frequency before and after the change for each user‑agent. Use free tools like CrawlerCheck or GoAccess to run reports without additional cost.

One operator lesson is that changes to robots.txt take up to 48 hours to be fully respected by crawlers that check the file periodically. Be patient and monitor the honeypot during this window.