---
title: "Is Cloudflare Blocking Googlebot and AI Crawlers on Your Site? How to Check"
slug: "cloudflare-blocking-googlebot-ai-crawlers-check-your-site"
type: "blog-post"
url: "https://automationbyexperts.com/blog/cloudflare-blocking-googlebot-ai-crawlers-check-your-site"
date: "2026-10-30"
category: "AI"
tags: ["cloudflare", "googlebot", "ai crawlers", "geo", "llms.txt", "seo audit", "ai visibility"]
read_time: 9
author: "Youssef Farhan"
---

# Is Cloudflare Blocking Googlebot and AI Crawlers on Your Site? How to Check

> Cloudflare's AI crawler defaults can refuse mixed-purpose bots like Googlebot at the network edge. Here is how to test your site, what the sources disagree on, and how to fix it.

**Short answer:** yes, Cloudflare's AI-crawler rules can keep Googlebot and other search bots out of your site, but mostly when someone has switched on AI-training blocking and the site is on Cloudflare. Reports on the change that took effect on 15 September 2026 disagree on exactly which sites get the new defaults, so the safe move is to check your own site now instead of assuming. This guide shows what changed, how to test whether you are affected, and how to fix it.

## What did Cloudflare change on 15 September 2026?

According to Help Net Security and TechCrunch coverage of Cloudflare's 1 July 2026 announcement, Cloudflare moved its default settings so that crawlers that serve a training or agent purpose are blocked on pages that show ads, while search crawlers stay allowed by default. The practical wrinkle, described by Search Engine Journal and several SEO agencies, is the "mixed-purpose" crawler: a bot such as Googlebot, Applebot or Bingbot that fetches pages for search and for AI features with the same user agent.

If a site blocks the training category, those mixed-purpose bots can be caught by the same rule even though the owner only meant to stop AI training. Search Engine Journal and Atwix both warn that this can block Googlebot. The block happens at the network edge, which matters for the next section.

One caveat before you panic: the sources do not agree on scope. Help Net Security describes the change as the default for new domains, while other write-ups say free-plan sites that never touched the setting inherit it, and one says the mixed-purpose rule only bites zones where training blocking was turned on. We did not read Cloudflare's original post, so treat the exact scope as unconfirmed and verify your own zone.

## Why does robots.txt not show the problem?

Robots.txt is a polite request that a crawler reads after it reaches your server. A Cloudflare rule acts before that: the request is refused at the network layer. Your robots.txt can say `Allow: /` for Googlebot while the bot receives a 403 or a challenge page. That is why the usual "robots.txt looks fine" check proves nothing here.

The signals that do show it are different: a drop in pages crawled in the Search Console crawl stats, new "Blocked due to access forbidden (403)" or "Server error" entries in the page indexing report, and URL Inspection failing to fetch a live page.

## How do I check whether my site blocks search and AI bots?

- **Open Cloudflare's AI crawler settings** for the zone (Security, Bots) and note whether blocking of AI training or agent crawlers is on, and whether the legacy "Block AI bots" toggle is active.
- **Run URL Inspection in Search Console** on your homepage and one deep page, using "Test live URL". A fetch failure with a 403 is the clearest warning.
- **Check the Crawl Stats report** (Settings, Crawl stats) for a falling request count after mid-September.
- **Request a page as the bot**. Sending a request with a Googlebot user agent from your own machine is not a reliable test, because Cloudflare verifies bots by IP, so rely on Search Console's live test instead.
- **Audit the whole site, not one URL**, because rules can differ by path, for example ad-supported pages only.

For step 5 you can automate the audit. The [SEO, GEO and AEO Audit actor](https://automationbyexperts.com/apify/seo-geo-aeo-audit) checks AI-crawler access and llms.txt, validates schema and returns a prioritized fix list per page, with no API keys. It tells you what a crawler-style request sees on each URL, which is the part Search Console only shows one page at a time.

| Check | Where | Bad sign |
| --- | --- | --- |
| Bot settings | Cloudflare dashboard | Training or "Block AI bots" on, with search not explicitly allowed |
| Live fetch | Search Console, URL Inspection | 403 or "page cannot be fetched" |
| Crawl volume | Search Console, Crawl stats | Steep drop after mid-September |
| Indexing | Search Console, Pages | New "403" or "server error" group |
| Site-wide view | Audit actor | AI-crawler access fails on many URLs |

## How do I block AI training without losing Google search?

The general approach the SEO write-ups recommend is to be specific instead of using a blanket switch:

- Keep search crawlers allowed explicitly, and review any rule that blocks "training" as a category when mixed-purpose bots share that category.
- For Google specifically, the Google-Extended token in robots.txt limits use of your content for Google's AI training without affecting normal search crawling, according to the SEO Journal summary of the issue.
- Use Cloudflare's verified-bot allow options or a custom rule for the search engines whose traffic you rely on, then re-test with a live fetch.
- Change one setting at a time and note the date, so a later dip in the crawl report can be tied to a cause.

This is general guidance from third-party sources, not legal or Cloudflare-official advice. Confirm the exact settings in Cloudflare's own documentation for your plan.

## Is blocking AI crawlers good or bad for AI search visibility?

It is a trade-off, and the evidence is thin. Blocking training crawlers keeps your content out of model training and, if Cloudflare's pay-per-crawl idea catches on, may one day earn money. But Cloudflare's own program was reported as still in beta with major AI companies not committed, so there is no proven payout yet. Blocking agent and answer-engine crawlers can also remove you from the pages those tools quote. We have no measurement showing how much visibility a block costs, so decide by what you want: protect content, or be cited.

If being cited matters, measure it instead of guessing. The [AI Brand Visibility Tracker](https://automationbyexperts.com/apify/ai-brand-visibility-tracker) asks ChatGPT, Perplexity, Gemini, Claude and Grok the questions your buyers ask and records whether your brand is mentioned, in what position, and which sources get cited. Run it before and after you change crawler settings to see whether the answers move. Our earlier guide, [is ChatGPT recommending your competitors](https://automationbyexperts.com/blog/is-chatgpt-recommending-your-competitors-ai-visibility), walks through reading those results.

## What about publishing an llms.txt file?

An llms.txt file is a plain-text index that tells AI tools what your site contains. It is a proposed convention, not a guarantee that any engine reads it, and it will not help if the crawler is refused before it reaches the file. Fix access first, then publish it. Our walkthrough [how to create an llms.txt file](https://automationbyexperts.com/blog/how-to-create-llms-txt-file) covers the format, and the [LLMs.txt Generator](https://automationbyexperts.com/apify/llms-txt-generator) builds llms.txt and llms-full.txt from your sitemap.

## What if I am the one scraping and now getting blocked?

The same shift affects people who collect data. More sites now return 403 or challenge pages to anything that looks automated. If you run scrapers, the practical response is to respect each site's terms and robots.txt, slow down, and use proper rendering and proxies for pages you are allowed to read. Our guide on [scraping a website that blocks bots](https://automationbyexperts.com/blog/scrape-website-that-blocks-bots-anti-bot-protection) covers the options. More tools for visibility work sit in the [SEO and AI visibility category](https://automationbyexperts.com/apify/category/seo-ai-visibility).

## Frequently asked questions

**Can Cloudflare block Googlebot?**Yes. Search Engine Journal and other SEO publishers report that when AI-training blocking is on, mixed-purpose crawlers such as Googlebot can be refused too, even if robots.txt allows them. Check with a live URL test in Search Console.

**Does robots.txt override a Cloudflare bot rule?**No. The rule acts at the network layer before the crawler can read robots.txt, so an Allow line does not help if the request is refused upstream.

**Did the 15 September 2026 change affect every Cloudflare site?**Not clearly. Sources disagree: some say new domains, some say free-plan sites that never changed the setting, and one says only zones with training blocking on. Verify your own zone instead of relying on a summary.

**How can I stop AI training but stay in Google search?**Allow search crawlers explicitly, review category-wide training rules, and consider the Google-Extended robots.txt token, which limits Google AI training use without affecting search crawling. Re-test with a live fetch after each change.

**How do I know if AI engines still mention my brand?**Ask them the questions your buyers ask and record the results over time. The AI Brand Visibility Tracker actor automates that across five engines.

## What should I do this week?

Start with the cheapest checks. Open the Crawl stats report and compare the weeks before and after 15 September: a sharp fall in requests is the first thing worth investigating. Next, run a live URL test on three page types, such as the homepage, a category page and an article, because rules that target ad-supported pages can treat them differently. If any returns a 403, open the Cloudflare bot settings and look at which category is switched on, then loosen it for the search engines you depend on and test again.

Then decide your policy on purpose. Write down which AI companies you are happy to be cited by, which you want to keep out, and who owns the setting. Many sites end up blocked only because a default was inherited and nobody reviewed it. Finally, schedule a repeat audit for next month, since bot lists and defaults keep changing and a block can reappear after a plan change or a new rule.

## Next step

Run an audit of your site today and check access before you tune anything else. [Try the actors on Apify](https://apify.com/fayoussef?fpr=youssef&fp_sid=blog), then use the tracker to see whether AI answers change.

Sources read: Help Net Security (2 July 2026), TechCrunch (1 July 2026), Search Engine Journal on AI crawler rules and Googlebot, and Atwix's pre-deadline check. We did not read Cloudflare's original announcement.

*Last updated: October 2026*

## Actors used in this guide

- [AI Brand Visibility Tracker - ChatGPT, Perplexity & Gemini](https://automationbyexperts.com/apify/ai-brand-visibility-tracker): Yes, there is software that tracks whether ChatGPT recommends your product over your competitors: this actor asks Cha…
- [LLMs.txt & llms-full.txt Generator for Any Website](https://automationbyexperts.com/apify/llms-txt-generator): It generates a spec-compliant llms.txt and llms-full.txt for any website in one click - crawling your sitemap or site…
- [SEO, GEO & AEO Audit - AI Search Readiness Checker](https://automationbyexperts.com/apify/seo-geo-aeo-audit): It audits any website for Google SEO and AI-search visibility in one run - returning 0–100 SEO, GEO and AEO scores pe…