Are manufacturers blocking AI crawlers? We read the robots.txt files of 113 manufacturers and distributors

Almost none of them are. On Oct. 26, 2026, we read the robots.txt files of 113 US manufacturers and industrial distributors, drawn from trade press honoree lists and association rosters. Of the 105 sites whose files we could read, six blocked any AI crawlers, and all six blocked only crawlers that collect training data. None blocked the bots that put a site into ChatGPT, Claude or Perplexity answers. None had shut out search engines by accident either.

For a mid-size manufacturer, we recommend always letting AI search and answer crawlers in, and letting the training crawlers in too unless you publish material you sell or license. Then check your CDN and website platform, where blocks now get switched on without anyone editing the file. The full data is free: download the dataset (CSV).

AmazonbotMixed use6 sites

CCBotTraining4

Applebot-ExtendedTraining4

BytespiderTraining4

meta-externalagentMixed use4

GPTBotTraining3

ClaudeBotTraining3

anthropic-aiOlder Anthropic token3

Google-ExtendedMixed use: training and grounding3

OAI-SearchBotAI search0

Claude-SearchBotAI search0

PerplexityBotAI search0

ChatGPT-UserFetches a page when a user asks0

Sites blocking each AI crawler, of 105 readable robots.txt files

Item	Value (sites)	Note
Amazonbot	6 sites	Mixed use
CCBot	4	Training
Applebot-Extended	4	Training
Bytespider	4	Training
meta-externalagent	4	Mixed use
GPTBot	3	Training
ClaudeBot	3	Training
anthropic-ai	3	Older Anthropic token
Google-Extended	3	Mixed use: training and grounding
OAI-SearchBot	0	AI search
Claude-SearchBot	0	AI search
PerplexityBot	0	AI search
ChatGPT-User	0	Fetches a page when a user asks

A bot counts as blocked when both the homepage and an inside page are disallowed for it. Six sites account for every block. Eight of the 113 sites refused our request and are left out. Source: Our analysis of robots.txt files from 113 US manufacturers and distributors, fetched Oct. 26, 2026.

How we built the sample, and what a robots.txt read can’t tell you

We drew the companies from public lists: 2024 and 2026 Top Shops honorees named by Modern Machine Shop (2024 list), processors from Plastics Technology’s 2025 plant visits, PMMI’s 2026 new members, Control Engineering’s 2026 System Integrator of the Year winners and CSIA members named in trade press, firms on Window + Door’s 2026 Top Manufacturers list, new PTDA distributor members, STAFDA leadership companies, and companies named in our other launch articles. We left out public companies and their subsidiaries. We did not verify headcounts, so a few of the private firms may be larger than 500 employees. Each company’s source list is in the dataset.

Between 9:11 and 9:13 p.m. Eastern on Oct. 26, we sent one plain request to each site’s main host for /robots.txt and /llms.txt, identified only as “PlantToPipeline research.” We read each file the way RFC 9309 tells crawlers to: use the group that names the bot, or the * group if none does; the longest matching rule wins, and Allow wins a tie. For 13 AI user agents we recorded whether each could fetch the homepage and an inside page.

Eight sites refused our request (seven with an HTTP 403 error, one by dropping the connection), which leaves 105 readable sites. A robots.txt rule is also only a request. The standard says its rules “are not a form of access authorization,” and Google’s robots.txt guide says they “cannot enforce crawler behavior.” A firewall or CDN can turn crawlers away whatever the file says, and that is invisible from outside. Fifty-five of the 113 sites are served through Cloudflare, which offers AI bot blocking as a setting. Cloudflare says that starting Sept. 15, 2026, new domains default to blocking training and agent bots on pages that display ads.

Which AI crawlers do what?

The operators’ own pages sort these bots by job: training data, AI search, or fetching a page because a user asked.

Bot	Operator	What it does, per the operator	Sites blocking it (of 105)
GPTBot	OpenAI	Training data for OpenAI models	3
ClaudeBot	Anthropic	Training data for Anthropic models	3
anthropic-ai	Anthropic	Older token; not on Anthropic’s current crawler page	3
CCBot	Common Crawl	Open repository of web crawl data; Mozilla found at least 64% of 47 language models it studied were trained on it	4
Applebot-Extended	Apple	Opts content out of Apple model training; a control token that does not crawl	4
Bytespider	ByteDance	No ByteDance page we could find; Cloudflare lists it as an AI crawler	4
Google-Extended	Google	Gemini training and grounding; a control token, no effect on Google Search	3
Amazonbot	Amazon	Improves Amazon products; may train Amazon AI models	6
meta-externalagent	Meta	Training AI models or indexing content	4
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search answers	0
Claude-SearchBot	Anthropic	Improves Claude’s search results	0
PerplexityBot	Perplexity	Perplexity search results; not used to train models	0
ChatGPT-User	OpenAI	Fetches a page when a user asks; robots.txt “may not apply”	0

The line that matters runs between the first nine rows and the last four. OpenAI’s crawler page says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links, and that each of its settings is independent, so blocking GPTBot keeps you out of training without touching search. Anthropic’s help page says blocking Claude-SearchBot stops it indexing your content for search.

Google works differently. AI Overviews and AI Mode use pages already in Google Search; Google’s guidance on AI features says there are “no additional technical requirements,” and the control is your robots.txt for Googlebot. Google-Extended has no effect on Search inclusion or ranking. It covers Gemini grounding as well as training, so blocking it is more than a training opt-out.

User-triggered fetchers mostly sit outside robots.txt. Perplexity says Perplexity-User “generally ignores robots.txt rules,” and Meta says its user-requested fetcher “may bypass” them.

What 113 manufacturers and distributors actually do

Industry	Sites checked	robots.txt readable	Block any training or mixed-use crawler	Block any AI search crawler	Serve a usable llms.txt
Metals and Machining	29	26	1	0	5
Plastics and Rubber	15	13	0	0	3
Industrial Machinery and Equipment	15	13	1	0	2
Automation and Industrial Tech	18	18	1	0	3
Building Products	19	19	0	0	3
Industrial Distribution	17	16	3	0	3
Total	113	105	6	0	19

Ninety-nine of the 105 readable sites let all 13 agents in, and 90 never name any of them, so the AI bots follow the same User-agent: * rules as Googlebot.

Name none of the 13 AI agents90 sites

Name AI agents and block none9

Block a training or mixed-use crawler6

Block an AI search or user-triggered crawler0

What 105 readable robots.txt files say about AI crawlers

Item	Value (sites)
Name none of the 13 AI agents	90 sites
Name AI agents and block none	9
Block a training or mixed-use crawler	6
Block an AI search or user-triggered crawler	0

Each site counts once. Four of the six blockers carry Cloudflare's managed block; the other two block Amazonbot in lists aimed mostly at SEO and data crawlers. Source: Our analysis of robots.txt files from 113 US manufacturers and distributors, fetched Oct. 26, 2026.

Four of the six blockers carry Cloudflare’s managed robots.txt block, with the line “Content-Signal: search=yes,ai-train=no.” A setting in the Cloudflare dashboard adds that block, and it disallows Google-Extended along with the pure training bots. One of the four sets GPTBot, ClaudeBot and Google-Extended to Allow inside the managed block. The other two blockers turn away Amazonbot and none of our other 12 agents, in lists that mostly block SEO and data-collection crawlers. That reads as bandwidth housekeeping more than an AI policy, though one of the two also blocks Google’s Vertex AI agent crawler.

Nine sites name AI agents and block none of them. Five Squarespace sites list 29 crawlers, GPTBot and ClaudeBot among them, in the same group as every other bot, which changes nothing. Path Robotics’ robots.txt names 29 search and AI agents above its * line and gives them one shared set of rules, starting with Allow. Standard Bots gives GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and others their own Allow lines; its * group also says only Allow, so nothing is lost.

We went looking for accidents, especially a leftover “Disallow: /” from a staging site, which tells every crawler to stay out. None of the 105 had one, and none blocked Googlebot. One site served a web page where its robots.txt should be, so it sets no rules. Distribution had the most blockers, three of 16, too few to call a trend.

Should my company block AI crawlers?

Never block the search and answer crawlers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those answers and does not keep you out of training. OpenAI ties training to GPTBot alone, and Perplexity says its search bot is not used to crawl content for AI foundation models. If your capabilities and data sheets are on the site to be found, these bots are how they get found; our study of which data sheets get cited in AI answers shows what gets picked up.

On training crawlers, we’d let them in. A mid-size manufacturer’s public site is marketing; you want models to know your name, what you make and what you’re certified for. What a model knows without searching comes from training data, and a block only affects future collection. Anthropic says a ClaudeBot block means the site’s “future materials” are excluded. And 94% of the sites we read allow every training crawler.

Block training crawlers if you publish material you sell or license, such as paid design guides or a reference library your distributors pay for. If something is truly sensitive, put it behind a login, since robots.txt is the wrong tool for that.

The robots.txt we recommend

For most manufacturers the right file says nothing about AI, because of a quirk of the standard: a bot named in its own group ignores everything under User-agent: *, so a search bot given only “Allow: /” will crawl your cart and login pages too.

# robots.txt for www.example.com
# Search engines, AI search, AI assistants and AI training crawlers
# all follow the rules below. We do not name AI bots on purpose:
# a bot named in its own group ignores the rules under "User-agent: *".

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/

Sitemap: https://www.example.com/sitemap.xml

Keep your own Disallow lines; the four above are examples. If you decide to opt out of training, add this group and leave the search and user bots alone:

# Opt out of AI model training. AI search and assistants stay allowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
Disallow: /

Google-Extended is left out on purpose because it also covers Gemini grounding. Add it only if you accept that trade.

In your industry

The recommendation holds in all six industries. Distributors with online catalogs should keep cart, account and search pages disallowed and product pages open; our guide to B2B e-commerce for distributors covers the product data those pages need. Job shops should keep the capabilities page on an open, crawlable URL, since an AI answer can quote it.

What to do Monday

1. Open yourdomain.com/robots.txt. Look for “Disallow: /” under “User-agent: *”, and for OAI-SearchBot, Claude-SearchBot, PerplexityBot or ChatGPT-User in a group that ends in “Disallow: /”. Fix either today.
2. If you use Cloudflare, check its AI crawler settings for edge blocks and for a managed block in robots.txt (look for the line “# BEGIN Cloudflare Managed content”). Ask your web host whether its firewall blocks AI crawlers.
3. After any platform or SEO plugin update, reopen robots.txt and llms.txt. Squarespace, Wix, Shopify, Yoast SEO and All in One SEO all wrote AI-related lines or files in our sample.
4. Decide on training crawlers on purpose and write the decision down. OpenAI says robots.txt changes take about 24 hours to reach its search systems.
5. Treat llms.txt as low priority. The llms.txt proposal, published by Jeremy Howard on Sept. 3, 2024, gives AI agents a summary of a site. It controls no access, and Google says its AI features need no AI text files. Nineteen of our 113 sites serve a usable one, 10 of them generated by an SEO plugin or site platform. Of the 24 sites that answered at /llms.txt, five served something broken: two empty files, two web pages and one word-processor document.
6. Then work on the pages these bots read, starting with Manufacturing SEO: what the pages that rank actually have.