Cloudflare's 15 September Change to AI Crawler Access, Measured

Cloudflare changed how it handles AI crawlers on 15 September 2026. A independent before/after measurement of 1,046 websites, published by monitoring company SeenSure, finds the change didn't move the way Cloudflare's own description of it implied — training crawlers were refused more often afterward, while search and AI-agent crawlers were refused dramatically less.

What was measured

SeenSure fetched each of 1,046 sites with ten identities in a fixed sequence — an ordinary browser, eight AI crawlers, then the same browser again — run before and after the 15 September change. 639 of the sites sat behind Cloudflare; 255 did not, serving as a control group to isolate whether any shift was actually caused by Cloudflare rather than general web drift. A repeated fetch by an ordinary browser, on both sides of the split, acted as a noise floor.

The non-Cloudflare control and the repeated-browser check both stayed flat across the measurement window, which is what makes the changes on the Cloudflare side attributable to the 15 September change specifically rather than background noise.

What changed

On sites behind Cloudflare, refusal rates moved in opposite directions depending on crawler type:

• Training crawlers were refused more often: ClaudeBot rose from 19.9% to 22.8%, GPTBot from 18.9% to 22.0%.
• Search and agent crawlers were refused far less often: all six of the remaining identities dropped 12.7 to 13.9 percentage points. The largest swing was OAI-SearchBot, from 16.9% down to 3.0%; the smallest was Perplexity-User, from 14.9% down to 2.2%.

Cloudflare's own stated scope for the change targeted training and agent crawlers, with search crawlers expected to hold flat. In SeenSure's data, search crawlers moved the most of any category, and agent crawlers moved in the opposite direction a tightening default would predict.

The baseline was already skewed before the change

Before 15 September, sites behind Cloudflare were already refusing AI crawlers roughly four to twelve times more often than sites that weren't — 25.2% versus 5.5% for the most-refused identity. Cloudflare-fronted sites made up 71.3% of the sample but accounted for 94.6% of all refusals recorded.

A later correction narrowed that gap once the sample was split by origin: 894 of the 1,046 hosts were sampled from Certificate Transparency logs of newly issued certificates (skewing toward young, newly registered domains), while 152 came from a European agency directory. On the agency-only subset, the Cloudflare-vs-not gap was 1.4x to 2.8x rather than four-to-twelve times — new domains are the population any CDN challenges hardest, since that's where abuse concentrates.

What the study won't claim

SeenSure is explicit about the limits of what a refusal rate can prove. 94.5% of the refusals in the dataset carried a cf-ray header, confirming Cloudflare was in the request path — but that doesn't prove Cloudflare's own bot rules made the decision, since a site's own origin server can return a 403 that gets proxied back through Cloudflare and looks identical from outside. The concentration of refusals on Cloudflare-fronted hosts (71.3% of the sample, 94.6% of refusals) is the part that's measurable from outside; who actually made each blocking decision isn't.

The published methodology also documents its own corrections in the open: an early version counted network errors as refusals by mistake (fixed 7 September), and an early draft mischaracterized the sample as "agency sites" when most of it was newly registered domains pulled from certificate logs (corrected 9 September).

Why it matters

Any site running behind Cloudflare with an interest in AI Overviews, AI Mode, or being cited by AI assistants should treat "training crawlers blocked more, search/agent crawlers blocked less" as the working hypothesis for what 15 September actually did — not the reverse, which is closer to how Cloudflare framed the change publicly. SeenSure says it plans to re-run the measurement in December to see if the pattern holds.

Sources

• What Cloudflare's 15 September change actually did to AI crawler access — SeenSure