# AI Crawlers on WordPress: Who Really Reads Your Site? URL: https://hacktheseo.com/ai-crawlers-wordpress/ Date: 2026-09-25 Last modified: 2026-09-23 Author: Eric Ibanez --- > On one WordPress site, 3,337 AI crawler visits in 30 days vs 229 from Googlebot, and 0 in Google Analytics. How to see them, count them and decide which ones to block. On a client’s WordPress site, an insurance broker, AI crawlers made **3,337 visits in 30 days**. Googlebot made **229**. In Google Analytics, none of these visits shows up: the counter reads 0. It isn’t a bug. Google Analytics counts visitors who run its script in a browser, and an AI crawler runs none: it downloads the text and leaves. The readers feeding ChatGPT, Claude, Perplexity and Meta AI are invisible in the tool most site owners check every morning. Here is how to see them, count them, and decide which ones to let in. #### The key points - **3,337 vs 229.** AI crawler visits vs Googlebot visits on the same WordPress site, over 30 days, measured server-side on 22 September 2026. - **0 in Google Analytics.** AI crawlers don’t execute JavaScript, so no browser-side analytics tool can count them. - **Meta leads** (1,202 visits), ahead of ChatGPT (688) and Perplexity (393). Meta alone crawls more than 5 times as often as Googlebot. - **“Claude-Web” doesn’t exist.** The name still sits in many robots.txt files, but it appears nowhere in Anthropic’s documentation. Blocking it does not block Claude. - **You can see them in your host’s access log**, or page by page with a plugin that counts them server-side. The live demo (3 min 15, in French): a ChatGPT visit seen server-side, the crawler ranking, and the two robots.txt traps. Everything it shows is written out below. ## What is an AI crawler? An AI crawler is a program that visits web pages on behalf of an artificial intelligence company. It identifies itself with a name sent in every request, the user-agent: GPTBot for OpenAI, ClaudeBot for Anthropic, meta-externalagent for Meta, PerplexityBot for Perplexity. They don’t all do the same job, and that changes everything when you decide what to do with them: - **Crawlers that collect data to train models**: GPTBot, ClaudeBot, meta-externalagent, CCBot (Common Crawl), Bytespider (ByteDance). - **Crawlers that read your page to answer a question**, sometimes at the very moment a user asks it: OAI-SearchBot and ChatGPT-User, Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User. Only the second group can produce an answer that cites you in front of a human. On another site we measure, **69% of AI crawler visits are for training and 31% for answering**. ## How many AI crawlers visit a WordPress site? Here is the log of a real site, an insurance broker running WordPress, over 30 days, read server-side. Crawler Company Visits in 30 days Meta AI (meta-externalagent) Meta 1,202 ChatGPT (GPTBot, OAI-SearchBot, ChatGPT-User) OpenAI 688 Perplexity Perplexity 393 Amazon Amazon 244 Claude Anthropic 212 **All AI crawlers** **3,337** Googlebot, for comparison Google 229 The robot activity screen of the Hack The SEO plugin on the measured site, 22 September 2026. The site name is hidden. (Server-side log, 1 site, 30 days, 22 September 2026) That’s **14 times more AI crawler visits than Googlebot visits**. One site is not an average: your ratio will be different, which is exactly why you should measure it instead of guessing. ## Why doesn’t Google Analytics show AI crawlers? Google Analytics, like Matomo or Plausible in their standard setup, records a visit when a small JavaScript snippet runs in the visitor’s browser. That snippet is what tells the analytics tool “someone is on this page”. An AI crawler does none of that. It requests the page from the server, receives the HTML, extracts the text and leaves. No browser, no script executed, no visit recorded. In the video, we trigger a visit from ChatGPT on the site: it shows up server-side within seconds, while Google Analytics real-time stays at 0. In practice, **the only reliable trace of an AI crawler is on the server**: in your host’s access log, or in a tool that reads requests before the page is sent. ## How to see which AI crawlers visit your WordPress site Three methods, from the most hands-on to the most comfortable. ### 1. Your host’s access log Every request your server receives leaves a line in it, with the date, the page requested and the crawler’s name. - **Download the** access.log **file.** On cPanel: Metrics, then Raw Access. On Plesk: Logs. On managed hosts (Kinsta, WP Engine, Cloudways, SiteGround): the Logs section. - **Count the AI crawlers**, on Mac or Linux, in the file’s folder: grep -Eio "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Meta-ExternalAgent|meta-externalfetcher|Amazonbot|CCBot|Bytespider" access.log | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn - **Compare with Google**: grep -ic googlebot access.log. One line from hacktheseo.com’s access log: Meta’s crawler, 22 September 2026 at 21:52. The rest of the log is blurred. ### 2. A plugin that counts them server-side The free [Hack The SEO plugin](https://hacktheseo.com/free-wordpress-seo-plugin/) recognizes AI crawlers the moment they request a page and counts them page by page, with no browser script, no account and no API key. That’s the screen in the capture above. The advantage over the raw log: you see right away which pages get read, and which never do. ### 3. Ask your AI If your WordPress site is connected to Claude through MCP, one sentence is enough: “Which 10 pages of my site did AI crawlers read the most over the last 30 days?”. Claude reads the counters and gives you the ranking. ## Why it matters to know who reads your pages For one very concrete reason: **a page no AI ever reads cannot be cited in its answers**. On the measured site, the ranking of pages most read by AI crawlers puts the homepage first, then the quote request page. That’s the page that brings in revenue, and it’s the one AI crawlers come to read. By contrast, an article published on 3 July got zero visits in 30 days. On another article of the same site, we counted 6 AI visits for 18 visits from classic SEO crawlers. Three decisions follow: - **Start with the pages that make you money**: check that they get read, and that they clearly answer the questions people ask an AI about your business. - **Find the pages that are never read**: internal links, sitemap, thin content. A crawler that can’t find a page can’t read it. - **Watch for changes**: a crawler that disappears overnight usually signals a block, not a loss of interest. ## Should you block AI crawlers in robots.txt? Blocking means making two different decisions, and many sites only make one without realizing it. - **Blocking training crawlers** (GPTBot, ClaudeBot, CCBot…): your pages stop being used to train models. You can still be cited. - **Blocking answer crawlers** (OAI-SearchBot, Claude-SearchBot, PerplexityBot…): you disappear from those AI answers. ### Trap #1: names that don’t exist Many blocklists, including the one we suggested ourselves until this week, block Claude-Web and anthropic-ai. [Anthropic’s documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) lists only three crawlers: ClaudeBot, Claude-User and Claude-SearchBot. We found this while preparing the video, and fixed it. Anthropic’s help page, checked on 22 September 2026: three crawlers, no Claude-Web. ### Trap #2: “User” agents may ignore your robots.txt ChatGPT-User, Perplexity-User and meta-externalfetcher fetch your page because a human asked a question. [OpenAI](https://platform.openai.com/docs/bots) and [Perplexity](https://docs.perplexity.ai/guides/bots) warn that robots.txt may not apply to these user-initiated visits. ### Trap #3: Google-Extended is not a crawler It’s a token. [According to Google](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), it doesn’t block any request and doesn’t remove you from AI Overviews: it only says your pages must not be used to train Gemini. ### The block to paste: no training, citations allowed User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Bytespider User-agent: MistralAI-Training Disallow: / Then open yoursite.com/robots.txt in your browser: that file is what crawlers read, not what your editor displays. ## Is your host already blocking ChatGPT? robots.txt is not the only gatekeeper. Before it sits your host’s firewall, and some block AI crawlers without telling you. We’ve seen it on sites hosted at IONOS, at PlanetHoster or behind Cloudflare. And [Perplexity states it in its documentation](https://docs.perplexity.ai/guides/bots): to appear in its results, you have to let its crawler in. The test takes two minutes: request the same page as a browser, then as an AI crawler. curl -sL -o /dev/null -w "browser: %{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36" https://yoursite.com/ curl -sL -o /dev/null -w "GPTBot: %{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://yoursite.com/ - **200 and 200**: nothing is blocking. - **200 for the browser, 403, 429 or 503 for the crawler**: that crawler is blocked. On Cloudflare, AI crawler blocking is switched off in the Bots section; elsewhere, email support with your test result. - **403 everywhere**: your own IP address is being refused. Run the test again from another connection. The test on a real site, 22 September 2026: the browser gets through, all three AI crawlers are refused. 3,337 AI crawler visits, 229 from Googlebot, and 0 in Google Analytics. AIs read your site far more often than you think, and no standard dashboard shows it. #### The limits of this measurement - **One site.** The table comes from an insurance brokerage site, over 30 days to 22 September 2026. It is not a web average. - **A visit is not a citation.** A crawler reading your page doesn’t prove an AI recommends you. But a crawler that never visits won’t cite you either. - **The 69% / 31% split comes from another measured site**, not the site in the table. ## Frequently asked questions ### How do I know if ChatGPT crawls my WordPress site? Look for GPTBot, OAI-SearchBot and ChatGPT-User in your host’s access log, or use a plugin that counts AI crawlers server-side. Google Analytics will never show them: these crawlers don’t run its script. ### Why doesn’t Google Analytics show AI crawlers? Because it counts a visit when a JavaScript snippet runs in a browser. AI crawlers download the HTML without running any script, so they leave no trace in browser-side analytics. ### Should I block GPTBot? Blocking GPTBot stops OpenAI from using your pages to train its models. It doesn’t remove you from ChatGPT search, which relies on OAI-SearchBot. If you want to keep being cited, don’t block OAI-SearchBot. ### Does Claude-Web still exist? No. Anthropic’s documentation only mentions ClaudeBot, Claude-User and Claude-SearchBot. Blocking Claude-Web or anthropic-ai in your robots.txt doesn’t block Claude. ### Does blocking Google-Extended remove me from Google’s AI Overviews? No. Google-Extended is a token that only opts your pages out of Gemini training. It blocks no request and has no effect on your presence in Search or in AI Overviews. ### Could my site be blocked for AIs without me knowing? Yes: some hosts and firewalls refuse AI crawlers by default. The curl test in this article checks it in two minutes: 200 for the browser and 403 for GPTBot means the crawler is blocked. ## Further reading - [GEO vs SEO: 6,175 keywords measured before and after AI Overviews launched in France](https://hacktheseo.com/geo-vs-seo-2/) - [llms.txt on WordPress: role, creation and setup](https://hacktheseo.com/llms-txt-wordpress-setup/) - [150 SEO and GEO statistics for 2026](https://hacktheseo.com/seo-geo-statistics/) - [The free Hack The SEO plugin for WordPress](https://hacktheseo.com/free-wordpress-seo-plugin/) **See your site’s AI crawlers, page by page.** The Hack The SEO plugin is free, with no account and no API key: [download the plugin](https://hacktheseo.com/download?utm_source=blog&utm_medium=article&utm_campaign=ai-crawlers-wordpress). Rather look at your site together? [Book 20 minutes with Eric](https://calendly.com/eric-ibanez-hts/20min). ### Our measurement - Hack The SEO, server-side log of crawler visits on an insurance brokerage WordPress site, 30 days to 22 September 2026 (robot activity screen of the free plugin). - Hack The SEO, training vs answer split of AI crawler visits on another measured site. - curl test on a real site, 22 September 2026. ### Crawler documentation - [Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, checked 22 September 2026](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - [OpenAI, Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) - [Perplexity, Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) - [Google Search Central, Google’s common crawlers (including Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) - [Meta, Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) **Eric Ibanez**, founder of Hack The SEO. Fifteen years in search, and our server logs open every day to watch GPTBot, ClaudeBot and PerplexityBot come through. Figures measured on 22 September 2026. --- Source: https://hacktheseo.com/ai-crawlers-wordpress/ Generated by Hack The SEO