AI crawlers: have French websites chosen to be visible in ChatGPT?

AI crawlers: have French websites chosen to be visible in ChatGPT?

I analysed the robots.txt of 9,891 .fr domains to answer that question. The answer fits in one number: 276 sites have written the name of the crawler that decides their presence in ChatGPT. The other 6,771 never have.

There is no such thing as “the ChatGPT crawler”. OpenAI runs several, with distinct roles, and each one is controlled separately. Blocking the one that collects training data does not remove you from the answers shown on screen.

Part of the web has made its call. 13.5% block training crawlers, and they do it methodically. That minority has occupied the entire public debate for three years.

The majority has not opened the file. This is not a reproach, it is a measurable observation, and it is the subject of this study.

The timing matters. Tomorrow, 15 September 2026, Cloudflare applies new default settings to AI crawlers. These figures are a snapshot of the French web taken just before the switch.

96% of French sites have never taken a position on their ChatGPT visibility

276 sites out of 7,047 name OAI-SearchBot in their robots.txt. That number does not say how many block it. It says how many have thought about it.

A robots.txt contains only what its author decided to put there. Naming an agent, whatever verdict follows, proves the question was opened. Not naming it proves the opposite.

Add the 21.1% of French sites with no robots.txt at all, and the picture is complete: the overwhelming majority of the French web has taken no position on its access to answer engines.

That silence is not neutral. With no directive, access is allowed by default, which is probably the right outcome for most sites. A default outcome is not a decision though: it changes the day a contractor adds a rule, a CMS regenerates the file, or a host modifies its settings. Nobody will notice, because nobody is watching.

In my view, this is the real finding of the study. The question that has stirred the industry for three years, whether to block AI, concerns 13.5% of the estate. The remaining 86.5% are not there yet. They do not know what their current configuration allows, and they have no way of knowing until they look at it.

What the analysis of 9,891 French sites reveals

78.9% of sites have a valid robots.txt, meaning 7,047 sites out of the 8,927 reachable ones. Every blocking rate below is calculated on that base.

The ranking of blocked crawlers draws a clear hierarchy. Training crawlers take the top spots, answer crawlers bring up the rear.

The gap runs four to one between GPTBot and OAI-SearchBot. Both crawlers belong to the same company, both are controlled by two neighbouring lines in the same file, and both end up treated in radically different ways.

The robots.txt is not the only lock. 10.2% of sites publish an llms.txt, and two sites across the entire corpus use the X-Robots-Tag: noai header. That last figure says everything about the real adoption of that signal.

Why blocking GPTBot does not remove you from ChatGPT

GPTBot and OAI-SearchBot do two different jobs. The first collects content that may train future models. The second feeds ChatGPT’s search function, the one that fetches pages and displays links under the answer.

OpenAI’s documentation is explicit: “Each setting is independent of the others”. Every setting is controlled separately, and a publisher can perfectly well allow OAI-SearchBot to appear in results while refusing GPTBot. The reverse is equally possible, and that is exactly what 727 sites in my sample do without knowing it.

The same trap exists at Google, in a worse form. Google-Extended is not a crawler, it is a control token: no request goes out under that name, Googlebot keeps crawling as usual, and the token simply states whether the retrieved content may train or ground Gemini. Google has revised that documentation to clarify the impact on search, precisely because the subject worried publishers who choose to block the token.

Direct consequence: blocking Google-Extended does not remove you from AI Overviews, which draw on the search index fed by Googlebot. 719 French sites, or 10.2%, are in that situation.

These sites did choose, and I checked

10.3% of French sites block GPTBot while leaving OAI-SearchBot open. Calling that a mistake would impute intent. So I measured rather than assumed.

A robots.txt names agents. The list of agents it names says what its author knew when writing it. Here is what those 727 files contain.

These files are not stale copy-paste. They name seven different training crawlers at the median, including Applebot-Extended and meta-externalagent, two identifiers that appeared after 2024. Nobody writes those names by accident. The anti-training policy is deliberate, kept current, and applied methodically to the whole family.

The same file says nothing about the answer family. 22.8% name at least one answer crawler, 6.9% name OAI-SearchBot. In other words, 93% of these sites have never taken a position on the crawler that decides their presence in ChatGPT’s answers.

This is neither an accident nor a complete choice. It is a policy built on a binary mental model, training against everything else, in which answer crawlers have no slot. They are not allowed by decision, they are allowed by absence of consideration.

In my view, that is good news poorly exploited. These 727 sites sit in the configuration I recommend: refuse training, accept citation. It holds as long as nothing moves. The day a contractor adds a blanket Disallow, a CMS regenerates the file, or a host changes its defaults, it unravels without anyone having decided. A position you never wrote down is a position you cannot defend.

One last figure confirms the direction of protection is understood: the opposite configuration, accepting training while refusing citation, covers 9 sites out of 7,047.

Those who decided distrust training, never traffic

Among the minority that has ruled, the dividing line is sharp. 824 sites write a User-agent: GPTBot group to refuse access. For Googlebot, that number drops to zero.

The 0.9% of sites where Googlebot is blocked are all caught by ricochet, through a restrictive User-agent: * that nobody wrote with it in mind. The calculation separates the two cases:

• Explicit decision: a group names the crawler. That is an assumed choice, documented in the file.
• Side effect: the crawler falls under a generic rule. Nobody ruled, the block arrives by accident.

Across the corpus, around 125 sites end up cut off from OAI-SearchBot by pure side effect, or 1.8%. Those sites made no decision about AI at all. They wrote a broad rule years ago, often to protect an admin area, and that rule now bites crawlers that did not exist at the time.

The GPTBot versus Googlebot contrast says something simple about risk appetite. Nobody cuts the branch that carries traffic. Everybody distrusts what looks like scraping. The problem is that the line between the two does not run where publishers place it.

222 sites block a crawler that no longer exists

One figure stopped me during the analysis. 222 sites explicitly name anthropic-ai in their robots.txt. That identifier circulated in 2023 blog posts; Anthropic no longer uses it. In the same corpus, only 62 sites block Claude-SearchBot, the crawler that actually feeds Claude’s answers.

Three times more sites block a ghost than the crawler in service. It is the most direct demonstration of what these files have become: copy-paste from three-year-old articles that nobody has reopened since. The same pattern repeats at Apple, where Applebot-Extended is blocked by 10.2% of sites against 2.1% for Applebot.

233 sites turn away a visitor coming to them

ChatGPT-User is blocked by 5.1% of sites. That crawler explores nothing on its own initiative: it fetches a page because a user asked ChatGPT to look at it. Refusing it amounts to shutting the door on someone knocking. OpenAI further states that robots.txt rules do not necessarily apply to it, which makes the block both hostile and uncertain.

News media are the only sector genuinely cutting themselves off

25.3% of media sites block GPTBot, against 4.7% of public-sector sites. Five times more. And the gap holds on the crawler that matters: 7.3% of media block OAI-SearchBot, against 1.3% of public-sector sites.

The media and public-sector intervals do not overlap. The gap holds statistically, it is not sampling noise.

The reading is consistent with where news publishers stand on the value of their archives. Media are the only sector to have run a collective debate on the subject, and it shows in their files. They are also the only ones blocking answer crawlers significantly, meaning they accept losing visibility in ChatGPT in exchange for control over their content.

At the other end, public-sector sites leave almost everything open. Town halls, schools, administrations: their mission is to distribute, not to monetise access. The result is logical, and it was worth quantifying.

What an AI crawler that cannot prove its identity sees

18.2% of sites refuse at least one request presenting itself as an AI crawler without being able to prove it. That number does not measure the blocking of real crawlers, and that needs saying before anything else.

I send these requests from an ordinary IP address. Serious firewalls validate crawlers through published IP ranges and reverse DNS. A firewall that refuses my request is therefore doing its job exactly right: it blocks an identity spoof. That is not a misconfiguration, it is good security, and the real GPTBot would most likely get through.

The figure keeps two uses, and neither consists of blaming infrastructure teams.

The first is practical. A site that refuses an unverified AI user-agent also refuses most GEO audit tools on the market, which hold no IP ranges validated by LLM vendors. If your visibility reports show empty pages with no explanation, that is often the cause, and it has nothing to do with your robots.txt.

The second concerns the gap between the two layers. At 10.8% of sites, the file allows while the network refuses an unverified request. At 6.3%, it is the reverse: the file forbids, the network lets through. That second case is the more interesting one, because it shows a site that believes it closed a door that stayed open for any client not reading the robots.txt.

Cloudflare appears on 2,930 sites in the corpus, far ahead of any other detected infrastructure. That is the main place where these rules get configured, often by a different team from the one writing the robots.txt.

Tomorrow, Cloudflare moves the decision out of your robots.txt

2,930 sites in my corpus sit behind Cloudflare, a third of the total. It is by far the leading infrastructure detected, ahead of Apache and Nginx servers exposed directly.

On 15 September 2026, Cloudflare replaces its binary switch with a three-behaviour classification: Search for crawlers that index, Agent for those acting in real time for a user, Training for those feeding model training. On pages detected as carrying advertising, Training and Agent become blocked by default while Search stays allowed.

The scope covers new customers, new sites of existing customers, and all current free accounts. Settings can be changed from the dashboard, before or after the date.

One trap is worth knowing: a crawler carrying several labels follows the strictest rule. Blocking Training can therefore block multi-purpose crawlers such as Googlebot, Bingbot or Applebot. The setting meant to protect your content can remove you from classic search.

The deeper change lies elsewhere. For twenty-five years, the ruling on crawler access lived in a text file at the root of the site, under the SEO team’s control. It is now moving up to the CDN account, managed by the infrastructure team. That is exactly the gap measured in the previous section, and it is about to widen.

In my view, the practical consequence is simple. The robots.txt remains a declaration of intent; it was never a lock. From tomorrow, the answer to “is my site accessible to AI” can no longer be read in one place. You have to check both layers, and check who controls each one.

38% of sites have no H1 in their raw HTML

Deliberate blocking is visible. Accidental blocking is far less so, and it affects more people.

One objection comes up often: AI engines would now render JavaScript, making the subject obsolete. Measurements say otherwise. The Vercel and MERJ analysis, covering more than 500 million crawler requests, finds no JavaScript execution in GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Bytespider or meta-externalagent. These crawlers sometimes download JavaScript files; they never execute them. The only exceptions are Googlebot, whose rendering infrastructure Gemini reuses, and Applebot.

Across 781 sites where I compared raw HTML against browser-rendered HTML, 38.4% expose no H1 title before JavaScript runs. 18.3% have their main content entirely dependent on JavaScript. 4.6% have their JSON-LD markup injected by script, therefore invisible to any crawler that does not render pages.

The median ratio between raw text and rendered text sits at 0.89. Most sites do fine. The tail of the distribution concentrates the problem, and those are often recent sites built on modern frameworks without server-side rendering.

Other signals tell the same story of a web poorly prepared for answer engines. Only 23% of home pages expose an update date, and barely 0.5% display it in plain text. 13.8% identify an author. 2.4% declare a Person node in JSON-LD. One site in two has no unique H1.

These figures cover the home page only, which is not an article page. They still give a sense of the average level of preparation: most French sites have put nothing in place to be understood by a machine composing an answer.

10.2% of French sites publish an llms.txt

911 sites in the corpus expose an llms.txt at their root. That rate surprises, yet it matches measurements published elsewhere: SE Ranking finds roughly one site in ten across 300,000 domains, Presenc.AI reports 10.13%, and Ahrefs reaches 28% across 137,210 domains receiving traffic. My 10.2% sits in the middle of that range.

The format proposes describing a site in Markdown for language models. The problem is not its adoption, it is its actual use. The Ahrefs study from May 2026 finds that 97% of those files received no request at all over the observed month, from crawler or human. SE Ranking finds no correlation between the presence of an llms.txt and citation frequency by LLMs.

The comparison with the study’s other figures is more interesting than the rate itself. Three times more sites publish an llms.txt than block OAI-SearchBot. Put another way, French publishers adopt a speculative format more readily than they reread the file that actually decides their accessibility.

A methodological note, because the objection is legitimate. Many CMS return a 200 status along with a home page when an inexistent file is requested, which mechanically inflates this kind of count. A first pass returned 1,025 files. By requiring plain text structured as Markdown, discarding anything containing HTML and anything resembling an error page, 114 disappear. The remaining 911 are real files.

The only documented use of the format sits away from answer engines: development assistants and documentation servers consume it actively, which explains its adoption in tech and SaaS.

What you can check in five minutes

Start with the simplest question: does your file name a single AI agent? If the answer is no, you have nothing to fix, you have a decision to make. Then, in this order.

• Your User-agent: * group: list what it forbids. Every rule applying to AI crawlers without your intent starts here.
• Answer crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. If one is blocked, you lose visibility, not control.
• Training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot. Blocking them is a defensible choice, provided you know it changes nothing about your presence in the answers.
• Your firewall: check its rules say the same thing as your file. At one site in ten, they contradict each other.
• Your Cloudflare dashboard, if you have one: since 15 September 2026, Training and Agent categories are blocked by default on ad-bearing pages of new domains.

The most frequent trap fits in one line. A User-agent: * group with Disallow: / on a staging environment left online, or a rule inherited from an old project. Nobody rereads that file, and it decides your presence in AI answers.

Once access is verified, the next question becomes measurable: do AI engines cite you, and on which queries. Access is a necessary condition; it has never been sufficient.

Methodology

Disclosure. This study is published by Cockpyt AI, which sells a paid tool for tracking visibility in AI answers. The aggregated raw data is downloadable, the collection method is described below, and the recommendations in this article can be implemented with a text editor and no tooling at all. Judge the figures, not the letterhead.

The study covers 9,891 .fr domains collected in September 2026. 8,927 responded, or 90.3%. The remaining 964 were discarded for DNS failure, TLS error or timeout. Every rate uses reachable domains as its denominator, and every percentage comes with its 95% Wilson interval.

The robots.txt analysis follows RFC 9309: a crawler obeys only the group whose token matches the longest prefix of its name, groups carrying that same token are merged, and failing that the generic group applies. The calculation systematically separates explicit decisions from side effects.

The network layer was measured on a subsample, with seven user-agents per domain, two seconds between requests and immediate abandonment of a domain on the first 429 code. It measures the reaction to an unverified AI user-agent and is never merged with robots.txt figures.

Sector segmentation relies on home page content: type declared in JSON-LD, cart markers, page title. Only confident classifications enter the calculation, which brings the sector base down to 2,964 sites. I tested two rule sets: the sector ranking stays identical in both, and the gap between media and public sector widens rather than narrows.

Stated limits. The corpus is a corpus of .fr domains, which does not guarantee every site is edited from France: 70.3% explicitly declare lang="fr". Structural signals cover the home page alone, which is not an article page. The network measurement uses no IP validated by LLM vendors, so it says nothing about the treatment reserved for authenticated crawlers. A robots.txt states what a site declares, never what a publisher thinks: the analysis of named agents is the best intent signal available to me, it is not proof. Finally, these figures are a snapshot at one date, and host default policies move fast.

Frequently asked questions

How do I know whether my site has already taken a position on AI crawlers?

Open your robots.txt and look for agent names: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended. If none appears, nobody has ever ruled on your site, and you are in the same case as 79.6% of French sites. Access is then allowed by default, which suits most sites, but it is not a held position.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data. The answers ChatGPT displays by fetching pages go through OAI-SearchBot, and user-triggered fetches go through ChatGPT-User. These are three independent settings.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended covers Gemini training and grounding. AI Overviews draw on the search index fed by Googlebot. Removing yourself from AI Overviews would mean blocking Googlebot, therefore disappearing from classic search as well.

Is a site without a robots.txt open to all crawlers?

Yes, the absence of a file counts as permission. 21.1% of French sites are in that case. A robots.txt returning an HTML error page under a 200 code is treated as absent by some crawlers and invalid by others, which makes behaviour unpredictable.

Is the llms.txt file useful today?

10.2% of French sites publish one. No major AI engine has confirmed using it for discovery or ranking, and 97% of these files receive no request at all. It is a bet on the future, not a measurable lever, and it never replaces a correct robots.txt.

Why does my GEO tracking tool see pages I thought were accessible?

Because the robots.txt and the firewall do not always say the same thing. At 10.8% of tested sites, the file allows access while the network layer refuses it. The firewall rule always wins, since it intervenes first.

Should I block training crawlers?

For most sites, yes, and that is the configuration I recommend: refuse training crawlers, allow answer crawlers. You keep visibility in ChatGPT, Claude or Perplexity without feeding model training for free. What matters is doing it knowingly, checking the answer crawler stays open.

Should I block ChatGPT-User?

No, in almost every case. That crawler does no automatic exploration, it fetches a page because a user asked ChatGPT to. Blocking it amounts to refusing an explicitly requested visit, protecting nothing in exchange. 5.1% of French sites do it.

What does the Cloudflare change of 15 September 2026 do?

Cloudflare now classifies crawlers into three behaviours: Search, Agent and Training. On pages carrying advertising, Training and Agent are blocked by default while Search stays allowed. The change applies to new customers, new sites of existing customers and free accounts. A multi-purpose crawler follows the strictest rule, so blocking Training can also block Googlebot or Bingbot.

How often should I reread my robots.txt?

Twice a year suits most cases, plus every redesign or hosting change. New crawlers appear fast, and a generic rule written five years ago now applies to agents that did not exist.

Sources

• OpenAI, Overview of OpenAI Crawlers, official documentation, accessed September 2026. developers.openai.com/api/docs/bots
• Search Engine Journal, Google Clarifies The “Google-Extended” Crawler Documentation, September 2024. searchenginejournal.com
• Search Engine Journal, OpenAI’s Crawler Docs Now List OAI-AdsBot For ChatGPT Ads, April 2026. searchenginejournal.com
• Vercel and MERJ, analysis of more than 500 million crawler requests, on the absence of JavaScript execution by dedicated AI crawlers. Reproduced and confirmed by several independent analyses in 2026.
• Ahrefs, llms.txt study across 137,210 domains, May 2026. ahrefs.com
• SE Ranking, llms.txt adoption study across 300,000 domains, November 2025, and absence of correlation with LLM citation frequency.
• Cloudflare, announcement of 1 July 2026 on classifying crawlers into Search, Agent and Training categories and the new default settings applicable from 15 September 2026.
• Cockpyt AI, proprietary data, September 2026 collection across 9,891 .fr domains.